System for coordinating user images in an artistic design

ABSTRACT

A computer system coupled to a digital image database with images having image attributes. The image attributes are automatically evaluated in order to find a compatible digital template from a digital template database. The templates each include a plurality of windows for displaying the digital images and each of the template include template attributes to automatically select a template whose attributes are compatible with at least one of the image attribute. The data base of digital images is also automatically searched for additional images having an attribute compatible with the template attributes and with the first one of the image attributes. A display screen coupled to the computer system displays the selected template and at least one other image.

CROSS REFERENCE TO RELATED APPLICATIONS

Reference is made to commonly assigned, co-pending U.S. patentapplications:

-   Ser. No. 12/512,070 by Ptucha et al. filed of even date herewith    entitled “Method for Matching Artistic Attributes of a Template and    Secondary Images to a Primary Image”;-   Ser. No. 12/512,056 by Bogart et al. filed of even date herewith    entitled “Artistic Digital Template for Image Display”;-   Ser. No. 12/512,061 by Bogart et al. filed of even date herewith    entitled “Method for Producing Artistic Image Template Designs”;-   Ser. No. 12/512,064 by Whitby et al. filed of even date herewith    entitled “Apparatus for Generating Artistic Image Template Designs”;-   Ser. No. 12/512,060 by Whitby et al. filed of even date herewith    entitled “Method of Generating Artistic Template Designs”;-   Ser. No. 12/512,067 by Ptucha et al. filed of even date herewith    entitled “System for Matching Artistic Attributes of Secondary Image    and Template to a Primary Image”;-   Ser. No. 12/512,057 by Bogart et al. filed of even date herewith    entitled “Coordinating User Images in an Artistic Design”;-   Ser. No. 12/512,054 by Shkurko et al. filed of even date herewith    entitled “Image Capture Device with Artistic Template Design”;-   Ser. No. 12/512,055 by Shkurko et al. filed of even date herewith    entitled “Image Capture Method with Artistic Template Design”;-   Ser. No. 12/512,077 by Ptucha et al. filed of even date herewith    entitled “Artistic Digital Template for Image Display”;-   Ser. No. 12/512,117 by Ptucha et al. filed of even date herewith    entitled “Method of Making an Artistic Digital Template for Image    Display”;-   Ser. No. 12/512,129 by Ptucha et al. filed of even date herewith    entitled “Context Coordination for an Artistic Digital Template for    Image Display”;-   Ser. No. 12/512,144 by Ptucha et al. filed of even date herewith    entitled “Processing Digital Templates for Image Display”; the    disclosures of which are incorporated herein by reference in their    entireties.

FIELD OF THE INVENTION

The present invention relates to computer systems, methods, and softwareprograms for use in making digital image products and digital imagedisplays. In particular, the present methods and apparatusesautomatically search for, find, evaluate, and arrange digital images ina digital image template according to programmed artistic designs.

BACKGROUND OF THE INVENTION

Personalized image collages, clothing, albums and other image enhanceditems are becoming increasingly more accessible as digital printingtechnologies improve. However, as personalized image bearing productshave become more accessible, consumers have become more discriminating.In particular, consumers now seek methods and systems that producecustomized image products in a more convenient, faster, seamless,automatic, and integrated manner. While becoming somewhat more common,many items for displaying and/or including embedded customized imagesare still considered novelties. Methods for recognizing image contentsthat fulfill a predescribed aesthetic appearance often fall short ofexpectations. For example, many products with customizable embeddedimages include photos of people. For this type of product, it would bedesirable to identify images that satisfy preselected artistic criteriaand/or image attributes, such as number of persons pictured, who ispictured, what zoom ratio, temporal aspects, clothing, background,season, facial expressions, hue, colorfulness, texture, and sharpness,etc. Because some artistic aesthetic elements work better for certainproduct formats, it would also be desirable if multiple image attributesincluding aesthetic criteria could be evaluated in parallel for a numberof images, and the images with the highest fitness score beautomatically determined by computer algorithm. Furthermore, customersatisfaction can be improved if the customer can select a favoriteimage, or several favorite images, to be included in an image enhancedproduct. Attributes from this favorite image can be extracted in realtime from the image itself, read from a file associated with the image,such as a metadata file, a DPOF file, or a file stored by a user, or itcan be generated in some other fashion, or even input directly by auser, and then be used in coordinating surrounding image and templatecontents. Conversely, attributes of a selected image product templatecan be used in coordinating which image or images from a user collectioncan be placed within the template. Furthermore, attributes of a contextor of an environment where the image will be displayed can be input to acomputer system and referenced to select appropriate templates andimages, or any combination of these algorithms can be integrated in acomputing system or method.

The problem in the prior art is the time consuming process requiringusers to manually search through many images and compose their ownartistic creations, for which they may not be capable.

SUMMARY OF THE INVENTION

An embodiment of the present invention includes a computer systemcoupled to a digital image database with images having image attributes.The image attributes are automatically evaluated in order to find acompatible digital template from a digital template database. Thetemplates each include a plurality of windows for displaying the digitalimages and each of the template include template attributes toautomatically select a template whose attributes are compatible with atleast one of the image attribute. The data base of digital images isalso automatically searched for additional images having an attributecompatible with the template attributes and with the first one of theimage attributes. A display screen coupled to the computer systemdisplays the selected template and at least one other image.

Another embodiment of the present invention includes a kiosk forcreating a coordinated image display. The kiosk includes a database ofdigital image files coupled to the kiosk and each digital image fileincludes an image and a plurality of image attribute values. Digitaltemplates are stored in a template data base and each have a pluralityof openings. The openings each have at least one required imageattribute associate with it. The images in the database of digital imagefiles are for placement in one or more openings in one of the digitaltemplates. One of the digital templates is automatically selected by thekiosk's evaluation of a compatibility of image attribute values torequired image attributes of one of the digital template's openings.

Therefore, one of the advantages of the present invention is automatingthe tedious steps for creating an image enhanceable product by acomputer system that operates upon a database of digital images usingartistic stored image attribute requirements stored as recipe segments.

These, and other, aspects and objects of the present invention will bebetter appreciated and understood when considered in conjunction withthe following description and the accompanying drawings. It should beunderstood, however, that the following description, while indicatingpreferred embodiments of the present invention and numerous specificdetails thereof, is given by way of illustration and not of limitation.Many changes and modifications may be made within the scope of thepresent invention without departing from the spirit thereof, and theinvention includes all such modifications.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a computer system for use in a preferred embodimentof the present invention.

FIG. 2 illustrates a user implementing a computer system in a preferredembodiment of the present invention.

FIG. 3 illustrates a digital template design in a preferred embodimentof the present invention.

FIG. 4 illustrates programming of required image attributes withweighted values.

FIG. 5 illustrates a method and results of a compatibility scorecalculation.

FIG. 6 illustrates image selection based on compatibility evaluation andimage adjustment.

FIG. 7 illustrates a flowchart for template selection.

FIG. 8 illustrates a flowchart for image selection.

FIG. 9 illustrates a flowchart for image selection.

FIG. 10 illustrates a flowchart for template selection.

FIG. 11 illustrates a flowchart for image selection.

FIGS. 12A-D illustrates algorithms for image searching and compatibilityscoring.

FIG. 13 illustrates examples of recorded metadata.

FIG. 14 illustrates examples of extracted metadata.

FIGS. 15A-B illustrates views of a frame for receiving a printedcompleted template.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 illustrates a first embodiment of an electronic system 26, acomputer system, for implementing certain embodiments of the presentinvention for automatically generating image enhanced products. In theembodiment of FIG. 1, electronic computer system 26 comprises a housing22 and a source of content and program data files 24 such as softwareapplications, template designs and recipes, image files, imageattributes and required image attributes, which includes various memoryand storage devices 40, a wired user input system 68 as well as awireless input system 58, and an output system 28, all communicatingdirectly or indirectly with processor 34. Although not shown processor34 is meant to illustrate typical processor system and chip componentssuch as instruction and execution registers, an ALU, various levels ofcache memory, etc. The source of program and content data files 24, userinput system 68, or output system 28, and processor 34 can be locatedwithin housing 22 as illustrated. In other embodiments, circuits andsystems of the source of content and program data files 24, user inputsystem 68 or output system 28 can be located in whole or in part outsideof housing 22. As an example, element 68 b illustrates a screen pointercontrol embodied as a mouse when located outside the housing 22 but canbe an embedded trackball when located within housing 22.

The source of content or program data files 24 can include any form ofelectronic, optical, or magnetic storage such as optical discs, storagediscs, diskettes, flash drives, etc., or other circuit or system thatcan supply digital data to processor 34 from which processor 34 can loadsoftware, template designs and recipes, derived and recorded metadata,image files, image attributes and required image attributes or deriveimages and image metadata for use in automatically forming an imageenhanced item. In this regard, the content and program data files cancomprise, for example and without limitation, software applications, astill image data base, image sequences, a video data base, graphics, andcomputer generated images, image attribute information associated withstill, video, or graphic images, and any other data necessary forpracticing embodiments of the present invention as described herein.Source of content data files 24 can optionally include devices tocapture images to create content data for use in content data files byuse of capture devices located at electronic computer system 20 and/orcan obtain content data files that have been prepared by or using otherdevices or image enhancement and editing software. In the embodiment ofFIG. 1, sources of content or program data files 24 includes sensors 38,a memory and storage system 40 and a communication system 54.

Sensors 38 are optional for particular embodiments of the presentinvention and can include light sensors, biometric sensors and othersensors known in the art that can be used to detect conditions in theenvironment of system 26 and to convert this information into a formthat can be used by processor 34 of system 26. Sensors 38 can alsoinclude one or more cameras, video sensors, scanners, microphones, PDAs,palm tops, laptops that are adapted to capture images and can be coupledto processor 34 directly by cable or by removing portable memory 39 fromthese devices and/or computer systems and coupling the portable memoryto slot 46. Sensors 38 can also include biometric or other sensors formeasuring involuntary physical and mental reactions. Such sensorsincluding, but not limited to, voice inflection, body movement, eyemovement, pupil dilation, body temperature, and p4000 wave sensors.

Memory and storage 40 can include conventional digital memory devicesincluding solid state, magnetic, optical or other data storage devices,as mentioned above. Memory 40 can be fixed within system 26 or it can beremovable and portable. In the embodiment of FIG. 1, system 26 is shownhaving a hard disk drive 42, which can be an attachable external harddrive, which can include an operating system for electronic computersystem 26, and other software programs and applications such as theprogram algorithm embodiments of the present invention, a templatedesign data base and a recipe data base, derived and recorded metadata,image files, image attributes and required image attributes, softwareapplications, and a digital image data base. A disk drive 44 for aremovable disk such as an optical, magnetic or other disk memory (notshown) can also include control programs and software programs usefulfor certain embodiments of the present invention, and a memory card slot46 that holds a removable portable memory 48 such as a removable memorycard or flash memory drive or other connectable memory and has aremovable memory interface 50 for communicating with removable memory48, if necessary. Data including, but not limited to, control programs,template designs and recipes, derived and recorded metadata, imagefiles, image attributes and required image attributes, softwareapplications, digital images and metadata can also be stored in a remotememory system 52 such as a personal computer, computer network, anetwork connected server, or other digital system.

In the embodiment shown in FIG. 1, system 26 has a communication system54 that in this embodiment can be used to communicate with an optionalremote input 58, remote memory system 52, an optional a remote display56, for example by transmitting image designs in the form of templatedesigns with or without merged images and receiving from remote memorysystem 52, a variety of control programs, template designs and recipes,derived and recorded metadata, image files data bases, image attributes,required image attributes, and software applications. Althoughcommunication system 54 is shown as a wireless communication system, itcan also include a modem for coupling to a network over a communicationcable for providing to the computer system 26 access to the network andremote memory system 52. A remote input station including a remotedisplay 56 and/or remote input controls 58 (also referred to herein as“remote input 58”) can communicate with communication system 54wirelessly as illustrated or, again, can communicate in a wired fashion.In a preferred embodiment, a local input station including either orboth of a local display 66 and local user input controls 68 (alsoreferred to herein as “local user input 68”) is connected to processor34 which is connected to communication system 54 using a wired orwireless connection.

Communication system 54 can comprise for example, one or more optical,radio frequency or other transducer circuits or other systems thatconvert data into a form that can be conveyed to a remote device such asremote memory system 52 or remote display 56 using an optical signal,radio frequency signal or other form of signal. Communication system 54can also be used to receive a digital image and other data, asexemplified above, from a host or server computer or network (notshown), a remote memory system 52 or a remote input 58. Communicationsystem 54 provides processor 34 with information and instructions fromsignals received thereby. Typically, communication system 54 will beadapted to communicate with the remote memory system 52 by way of acommunication network such as a conventional telecommunication or datatransfer network such as the internet, a cellular, peer-to-peer or otherform of mobile telecommunication network, a local communication networksuch as wired or wireless local area network or any other conventionalwired or wireless data transfer system.

User input system 68 provides a way for a user of system 26 to provideinstructions to processor 34, such instructions comprising automatedsoftware algorithms of particular embodiments of the present inventionthat automatically generate artistic coordinated image displaysaccording to template recipes. This software also allows a user to makea designation of content data files, such as selecting image templatesand designating primary images, to be used in automatically generatingan image enhanced output product according to an embodiment of thepresent invention and to select an output form for the output product.User controls 68 a, 68 b or 58 a, 58 b in user input system 68, 58,respectively, can also be used for a variety of other purposesincluding, but not limited to, allowing a user to arrange, organize andedit content data files, such as coordinated image displays and imagetemplates, to be incorporated into the image enhanced output product,for example, by incorporating image editing software in computer system26 which can be used to override automated image enhanced outputproducts generated by computer system 26, as described below in certainembodiments of the present invention, to provide information about theuser or audience, to provide annotation data such as voice and textdata, to identify characters in the content data files, and to performsuch other interactions with system 26 as will be described later.

In this regard user input system 68 can comprise any form of devicecapable of receiving an input from a user and converting this input intoa form that can be used by processor 34. For example, user input system68 can comprise a touch screen input 66, a touch pad input, a 4-wayswitch, a 6-way switch, an 8-way switch, a stylus system, a trackballsystem, a joystick system, a voice recognition system, a gesturerecognition system, a keyboard 68 a, mouse 68 b, a remote control orother such systems. In the embodiment shown in FIG. 1, electroniccomputer system 26 includes an optional remote input 58 including aremote keyboard 58 a, a remote mouse 58 b, and a remote control 58 c.Remote input 58 can take a variety of forms, including, but not limitedto, the remote keyboard 58 a, remote mouse 58 b or remote controlhandheld device 58 c illustrated in FIG. 1. Similarly, local input 68can take a variety of forms. In the embodiment of FIG. 1, local display66 and local user input 68 are shown directly connected to processor 34.

As is illustrated in FIG. 2, computer system 26 and local user input 68can take the form of an editing studio or kiosk 70 (hereafter alsoreferred to as an “editing area 70”), although this illustration is notintended to limit the possibilities as described in FIG. 1 of editingstudio implementations. In this illustration, a user 72 is seated beforea console comprising local keyboard 68 a and mouse 68 b and a localdisplay 66 which is capable, for example, of displaying multimediacontent. As is also illustrated in FIG. 2, editing area 70 can also havesensors 38 including, but not limited to, camera or video sensors 38,audio sensors 201 and other sensors such as, for example, multispectralsensors that can monitor user 72 during a user or production session.

Output system 28 (FIG. 1) is used for rendering images, text, completedor uncompleted digital image templates, or other graphicalrepresentations in a manner that allows an image enhanceable item to beconverted into an image enhanced product or display, such as a digitalframe, LCD display, photo album, or collage. In this regard, outputsystem 28 can comprise any conventional structure or system that isknown for printing, displaying, or recording images, including, but notlimited to, printer 29. For example, in other embodiments, output system28 can include a plurality of printers 29, 30, wherein processor 34 iscapable of printing to a plurality of printers or a network of printers.Each printer of the plurality of printers can be of the same or adifferent type of printer than at least one other of the plurality ofprinters, and each printer can produce prints of the same or a differentformat from others of the plurality of printers. Printer 29 can recordimages on a tangible surface, such as on, for example, various standardmedia or on clothing such as a T-shirt, using a variety of knowntechnologies including, but not limited to, conventional four coloroffset separation printing or other contact printing, silk screening,dry electrophotography such as is used in the NexPress 2100 printer soldby Eastman Kodak Company, Rochester, N.Y., USA, thermal printingtechnology such as in thermal printer 30, drop on demand ink jettechnology and continuous inkjet technology. For the purpose of thefollowing discussions, printers 29, 30 will be described as being of atype that generates color images. However, it will be appreciated thatthis is not necessary and that the claimed methods and apparatusesherein can be practiced with printers 29, 30 that prints monotone imagessuch as black and white, grayscale or sepia toned images.

In certain embodiments, the source of content data files 24, user inputsystem 68 and output system 28 can share components. Processor 34operates system 26 based upon signals from user input system 58, 68,sensors 38, memory 40 and communication system 54. Processor 34 caninclude, but is not limited to, a programmable digital computer, aprogrammable microprocessor, a programmable logic processor, a series ofelectronic circuits, a series of electronic circuits reduced to the formof an integrated circuit chip, or a series of discrete chip components.

As used herein, an embodiment of an image enhanceable item 300, such asthe example frame shown in FIG. 3, can include anything that has atangible surface 302 with which an image can be formed, displayed,popped in, inserted, attached, adhered, located, embedded, placed orotherwise provided. In addition, an image enhanceable item can beextended to include non-tangible items such as holograms or otherdigitally controlled virtual displays. An image enhanceable item oritems can be saved on various recording media for playback on digitaldisplays, projectors, home theater systems as slideshows. For exampleand without limitation, an image enhanceable item 300 can be formed invarious shapes such as heart shaped, a person's profile, the shape of ageological area, or any other shape. The tools and techniques forgenerating image templates of various shapes, in general, are well knownin the art and are not described further herein as to their initialcreation. It can also take the form of a collage, photo book, scrapbook, photo calendar, mug, stein, cup, stemware, jewelry, tile, mosaic,home décor, mouse pads, pillowcases, pen & pencil holders, a simulatedor actual brushstroke image on canvas, a photo-realistic image on acanvas, a keepsake box, a fleece blanket, coasters, frames, ornaments,round ornament, snowflake ornament, filigree ornament, pewter ornament,holiday ornament set, annual ornament set, playing cards, puzzle, teddybear or other stuffed animal, wall paper, packaging, apparel &accessories, including, but not limited to, a T-shirt, a tie, a totebag, apron, baby onesie, performance shirt, and/or frame, matte andimage combinations and collages, mailing labels, gift tags stamps, orany other tangible item.

In addition, other embodiments of an image enhanceable item are alsocontemplated. Embodiments of the present invention also includeelectronic displays such as picture frames, digital store frontdisplays, digital projectors, score boards such as in a stadium, orelsewhere, billboards, and interactive displays. Such displays havespatial and temporal components which are described below. The temporalcomponent can be used to display transient information, advertising,artistic content, and can be saved for later display.

Thus, the image enhanceable item 300 as shown in FIG. 3 can be anelectronic digital display, or template. In this form, the electronic“virtual” display as shown can realistically represent an actual frameavailable at a retail location that provides a kiosk that includes thesoftware and/or computer system of the present invention. A useractivates the kiosk to automatically generate an artisticallycoordinated image display, such as the “virtual” template with virtualwindows or openings shown in FIG. 3 displayed on screen 66 of computersystem 26, with the user's images merged therein. The user can selectthis virtual template image to be printed on attached printer 29. Aprint of this electronic virtual template display, with the user'sdigital images located in the electronic virtual windows therein, canthen be inserted or located in the actual frame available at the retaillocation so that the printed images align with actual physical openings,or windows, in the frame. The “virtual” openings 306, 308, and 310 onthe display can be used to display images or videos or a combination ofboth. The color and texture of an electronically displayed template 312can be selected to resemble any material. In a preferred embodiment, thetexture can be selected to resemble an actual tangible picture framewhich a user can purchase. This allows the user to experience the actuallook and feel of the frame containing his or her images placed in thewindows before printing the completed template design. The user canselect alternate templates to be displayed with the same images disposedtherein or with a new set of images, under control of the softwareembodiments of the present invention being run on the computer system26, as described previously. The display template 300 shown in FIG. 3can be formed in any shape, such as an oval, diamond, trapezoid, or anyother regular or irregular shape. The openings 306, 308, and 310 in theelectronic display format can also be shaped and patterned, in any formor shape and in any number and location within the template. Thetemplate described above and any images or videos disposed therein canbe displayed on an electronic display 66 coupled to the computer system26 as described above. The background 312 of the template can includedecorative background images 320, 330, or videos to enhance orcomplement the look and feel of the images or videos disposed in otherones of the openings. In such a format, the frame would consist of adigital picture frame circuit controlling the display in one or more ofwindows 306, 308, 310, and having remaining openings therein to displayimage prints. Thus, one or more of the windows 306, 308, 310 would be anelectronic LCD or OLED display, or other display technology, and theremaining windows used for disposing printed images. Any one or more ofthe openings 306, 308, 310 can include single images or single videoswhile other openings can display changing images or videos. Such imagesor videos can include transitions (a still or motion image that changesover to another still or motion image, which can be the same ordifferent from a current still or motion image) that can be separatelytimed or can occur simultaneously with transitioning images or videosdisplayed in other openings. An advantage of certain embodiments of thepresent invention is that various combinations of images, videos, framedesigns, etc. can be displayed, under user control, in alternative formsas a virtual template on a display of computer system so that a user canselect a desired look for an image enhanced product. If the electronicvirtual template as shown in FIG. 3 is used as a template for display ona scoreboard at a stadium event, for example, the openings can be usedto display live feeds from any number of selectable cameras situated indifferent locations at the stadium.

It is the intent of particular embodiments of the present invention toset forth a language, method, program, or mechanism for an artist todescribe his/her aesthetic intent for each template window or templateopening and for the image display overall, and for this intent to befulfilled in an automatic manner on images in real time using a computersystem such as illustrated in FIGS. 1-2. The description, program,mechanism or language for each template window opening has been referredto as a recipe, formula, or as required image attributes, which aredescribed in more detail below. These formulas can be designed byartists, designers, celebrities, sports stars, or others; transmittedand/or loaded and stored in computer system storage 40 for repeated useby any number of users of computer system 26. The formulas for theopenings can be independent of the template used. An overall templaterecipe comprising a number of window recipes, however, is typicallydesigned for an overall look and feel of a template and its mergedimages, and can be stored a template recipe file. In a preferredembodiment, the template recipe file is named so that the aestheticeffect of the template recipe is evident to users. For example, filename “Fall Colors” can identify for a user the look and feel that willbe automatically generated by selecting and activating such a templaterecipe on computer system 26.

As another example of the operation of an embodiment of the presentinvention, with a template containing two window openings, we have leftright required image attributes or left and right recipes or left andright formulas. The recipe for the opening on the left may dictate thatwe have a colorful non-person main subject image which fills the openingat a 120% zoom. The recipe for the opening on the right may dictate thatwe have a landscape image with two people; both facial poses aredirected slightly to the left, each face having a 20% fill factor andboth faces be offset to the left by 20%. In order to fulfill the recipeseach of a plurality of computer system accessible images is evaluated,via programmed computation as described below, based on a fitnessmeasure that calculates an image's compatibility with an opening'srequired image attributes, for each image with respect to satisfying therecipes for the virtual frame openings. Thus, some images might bettersatisfy a template opening recipe than another image as indicated by ahigher score, evaluation, or compatibility calculation. The images withthe highest score, automatically obtained by executing the programming,for each window opening, as against a number of stored images, isautomatically chosen to fill each window opening without requiring userintervention in most instances. For most cases, the user merely needs toprovide a number of digital images to the computer system, e.g.attaching a flash drive at slot 46 that contains some or all of theuser's image files and, in certain, but not all, embodiments of thepresent invention, the user also selects a favorite image or imageswhich is then used by the computer system as a basis to automaticallyselect remaining images to fill out a template display and/or toautomatically select a template. The user may also provide a networkserver location to the computer system where some or all the user'simages are located. The network connected computer system will thenretrieve these images for use in generating a completed image displaytemplate.

Alternatively, the scoring computation can be programmed such that anoptimum arrangement of images in the window openings is one thatgenerates an overall highest score for the entire template. Anarrangement that generates the highest overall score is chosen.Identifiable or measureable characteristics that might be included toaffect score calculations include, but are not limited to, identity ofmain subject, main subject size, number of faces, face size, facialpose, facial expression, age/gender of face, dominant color, textureanalysis, object match, and time stamp. Time stamps can be effective tosearch for and find images captured during a particular season. Thesevarious characteristics of images are referred to herein as metadata ormetadata tags, and are described in more detail below. Metadata can berecorded in association with digital images as derived (extracted)metadata, or as static (recorded) metadata. Typically, users can providerecorded metadata directly by operation of a camera or by later openingimage files on a computer system and providing recorded metadata forstorage at that time. Almost all digital cameras associate and storemetadata with image files captured by the camera, such as time stamps,focal settings, file names, GPS location data, etc. Extracted metadatais obtained by processing image data files, either by a camera'scomputer system or other external hand held, or workstation, kiosk, orlaptop computer system, using programmed algorithms for extractingcharacteristics in the image. A reference which explains computerizedfacial age estimation of persons in digital images is Lobo and Kwon,“Automatic Feature Detection and Age Classification of Human Faces inDigital Images”, U.S. Pat. No. 5,781,650. A reference that discussescomputer executed facial expression determination is described in Yin,Loi, Xiong, “Facial Expression Analysis Based on Enhanced Texture andTopographical Structure”, Conference Proceedings—IEEE InternationalConference on Systems, Man and Cybemeteics, v1, P586-591, 2004. Areference that describes digital gender estimation is Tivive,Bouzerdoun, “A Gender Recognition System Using Shunting InhibitoryConvolutional Neural Networks”, IEEE International Conference on NeuralNetworks—Conference Proceedings, p 5336-5341, 2006. These threereferences are incorporated herein by reference in their entirety.“Recipes” are discussed below with respect to preferred embodiments ofthe present invention.

A majority of keepsake photographic memories contain pictures of people,and, as such, people are often the main subjects in images and,therefore, often may be critical in fulfilling recipe requests. Usingcomputer methods described in U.S. Pat. No. 7,508,961B2 by Chen,Nicponski, and Ray, or P. Viola and M. Jones, “Rapid object detectionusing a boosted cascade of simple features,” in Computer Vision andPattern Recognition, 2001, Proceedings of the 2001 IEEE Computer SocietyConference, 2001, pp. I-511-I-518 vol. 1, or Schnieiderman, H.Schneiderman, “Feature-centric evaluation for efficient cascaded objectdetection,” in Computer Vision and Pattern Recognition, 2004.Proceedings of the 2004 IEEE Computer Society Conference, 2004, pp.II-29-II-36 Vol. 2, the size and location of each face can be foundwithin each image. These three documents are incorporated by referenceherein in their entirety. Viola utilizes a training set of positive faceand negative non-face images. Then, simple Haar-like wavelet weakclassifier features are computed on all positive and negative trainingimages. While no single Haar-like feature can classify a region as faceor non-face, groupings of many features form a strong classifier thatcan indeed be used to determine if a region is a face or not. Thisclassification can work using a specific size window. This window isslid across and down all pixels in the image looking for faces. Then thewindow is enlarged and larger faces are scanned in the image. Theprocess repeats until all faces of all sizes are found in the image.Because this process can be quite compute intensive, optimizations suchas an integral image and cascades of weak classifiers makes the processwork faster. Not only will this process find all faces in the image, itwill return the size of each face. Similar techniques can be used tofind the location and size of any non-deformable object such as people,cakes, balloons, graduation caps, cars, boats, and other desirableobjects.

Once a face is found, neural networks, support vector machines, orsimilar classifying means can be trained to locate specific featuressuch as eyes, nose, and mouth; and then corners of eyes, eye brows,chin, and edge of cheeks can be found using geometric rules based uponanthropometric constraints such as those described in DeMenthon, DanielF, Davis, Larry S., “Model Based Pose in 25 Lines of Code”, Proceedingsfrom the Image Understanding Workshop, 1992. Active shape models asdescribed by Cootes, T. F. Cootes, C. J. Taylor, D. H. Cooper, and J.Graham in “Active shape models—their training and application,” ComputerVision and Image Understanding, vol. 61, pp. 38-59, 1995, can be used tolocalize all facial features such as eyes, nose, lips, face outline, andeyebrows. These two publications are incorporated by reference herein intheir entirety. Once all features are found, it is possible to determineif eyes/mouth are open, or if the expression is happy, sad, scared,serious, neutral, or if the person has a nice smile. Determining poseuses similar extracted features. A. Savakis, M. Erhard, J. Schimmel, andJ. Hnatow in “A multi-camera system for real-time pose estimation,”Bellingham Wash., 2007, developed a geometric model that adhered toanthropometric constraints. This document is incorporated by referenceherein in its entirety. With pose and expression information stored asmetadata for each image, the present invention can give the artist morefunctionality to define increasingly nuanced image enhanced products byallowing the artist to specify in a window recipe, for example, a personlooking to the left with a serious demeanor.

In many instances there are no people depicted in an image, but there isa main subject that is not a person or that does not contain arecognizable face. A main subject detection algorithm, such as the onedescribed in U.S. patent application Ser. No. 09/223,860, filed Dec. 31,1998, which is incorporated herein by reference in its entirety, can beused to search for and find an image to fill each opening in anaesthetic manner according to stored recipes. Such a recipe mightrequire a particular object to be present in a an image which issearchable by virtue of that object being identified in the metadataassociated with an image. The metadata for images in a data base ofimages is searched and those images whose metadata satisfies the searchare found and evaluated according to programmed evaluation procedureswhich are typically defined as calculations to provide a compatibilityindex as between a found image and a template opening's required imageattributes.

Other exemplary embodiments of such algorithms involve segmenting theimage into a few regions of homogeneous properties such as color andtexture. Region segments can be further grouped into larger regionsbased on similarity measures. Regions are algorithmically evaluated fortheir saliency using two independent yet complementary types of saliencyfeatures—structural saliency features and semantic saliency features.The structural saliency features are determined by measureablecharacteristics such as location, size, shape and symmetry of eachregion in an image. The semantic saliency features are based uponprevious knowledge of known objects/regions in an image which are likelyto be part of foreground (for example, statues, buildings, people) orbackground (for example, sky, grass), using color, brightness, andtexture measurements. For example, identifying key features such asflesh, face, sky, grass, and other green vegetation by algorithmicprocessing are well characterized in the literature. The data for bothsemantic and structural types can be integrated via a Bayes net asdescribed by Russell and Norvig, “Artificial Intelligence—A ModernApproach,” 2^(nd) Edition, Prentice Hall, 2003, to yield the finallocation of the main subject. This document is incorporated by referenceherein in its entirety. Such a Bayes net combines the prior semanticprobability knowledge of regions, along with current structural saliencyfeatures into a statistical probability tree to compute the specificprobability of an object/region being classified as main subject orbackground. The main subject detection algorithm provides the locationof a main subject and the size of the subject as well.

Using recipes, an artist may design and store in the computer systemmemory creative required image attributes for a number of templateopenings defining an image template recipe, thereby creating severalaesthetic or practical looks for a single image enhanced item. A retailstore that provides the tools described herein for customer use andpurchase can choose to roll out new looks for each season, holiday, orother occasions. A store that provides these product services canlicense recipes to or from famous artists or celebrities. An aestheticlook can also be achieved by programmed computer system analysis of auser's images or from a user's selection of a primary or favorite imageor images, determining a theme or pre-determined look from image toimage, and then picking a template that is complementary to that themeor look. A preferred embodiment of the present invention contemplates aninitial step of characterizing images in order to select a templatebased on those characterizations, and also contemplates an initialselection of a template in order to identify appropriate images forinclusion. In each case, recorded image metadata is used, or isgenerated by extraction and stored with the images, and is compared withmetadata stored with an image template to evaluate, via an evaluationprocedure, compatibility between them. A preferred embodiment of thepresent invention extracts characteristics of a user's designatedfavorite image or images and incorporates those characteristics inwindow recipes for remaining openings in a template.

Another embodiment of the present invention includes computer systemanalysis of digital images for information content such as number ofpeople and/or their expressions and/or color content, then the imagesare computationally evaluated against selected aesthetic intents asdefined by a template or window recipe, where such artistic or aestheticintents include specific design layouts, each layout being created inadvance by an artist or someone skilled in the art of making an imagelayout pleasing or enticing. For a tangible consumer product, such as acollage, frame, or photo book, the intent may include making somethingpleasing, creative, fun, inspirational, romantic, or to invoke pastmemories. For transient displays, such as the electronic displaysdescribed above, the intent can be the same as for image enhanceableconsumer products, such as digital picture frames or cameras, but may bequite different for commercial applications using electronic displays ofvarious sizes. The following are examples of commercial applications: adigital storefront may be continually updating its product line in anattractive and enticing manner, an embodiment of the present inventionmay include a large or small digital display in a storefront window. Analternative embodiment may include a user interactive display. A newsbillboard intent may be to group images by subject and relevance. Asports venue billboard or scoreboard may include displaying picturesthat work well together gathered from a continual scanning and capturingof images of the attending crowd or other scenes in or near the sportsvenue. An interactive display can have a mix of live video frames andpre-existing digital libraries and update its content based upon userinput, where the user input can be received via a traditional mouse andkeyboard, or may be haptic interfaces, or the display may update itscontent automatically by detecting a user's facial expression and useractivity, and selecting those kinds and types of images for display thatelicit a particular electronically detected response. The aestheticintent that scores highest, or works best for that particular image setis chosen automatically for display or inclusion in an image enhanceableproduct. Scoring is symmetric in that a template can be selected inadvance and the highest scoring images are included, or one or moreimages can be selected in advance, and then the highest scoring templateis selected based on its aesthetic compatibility with the selectedimages. Alternatively, when creating image enhanceable consumerproducts, the top n highest scoring images/templates/recipes can bepresented to the customer on a computer system display, who in turnpicks the image/template/recipe of his/her choice and communicates sucha choice to the computer system via the user input controls.

Given an aesthetic look defined by a recipe, filling each window in animage enhanceable item can be done in at least two ways. If there are nwindows and n images, a preferred embodiment of the method of thepresent invention can include computing all possible combinations of allimages contained in an image data base in each window. If there are nwindows in a virtual image template and m images in an image database,where m>n, a preferred embodiment of the present invention could havemany variations to score. In the latter condition, a preferredembodiment of the invention is to place the image with the highest scoreinto each window. In this example, each image is scored by programmedcomputation against each window recipe individually, and the image withthe highest compatibility score for a particular window gets assigned orselected to that particular window. If an image is calculated to have ahighest score that is identical in greater than one window, the windowwith the higher weight is considered more important and will be used todisplay the high scoring image. Template window weighting is describedin more detail below where priority of template windows correspond tothe numbering of the windows in a template. If multiple images arescored equivalently for a single window opening, metadata fields thatrecord the number of times an image was accessed can be used to break atie. To resolve ties, an algorithm that randomly selects an image fromthe group can be used. The use of weighted values for images and fortemplate openings is described in more detail below.

Another common template recipe generation and fulfillment technique isimage splitting. Image splitting is the process of taking a singlecustomer image, and spanning that one image over 2 or more windowopenings. For example, the program can receive as input a single pictureof a bride and groom. An application of a particular recipe will resultin the bride being placed in the left window with the groom in the rightwindow. An artist will be able to define and store recipe requirementsfor the size, location, and spacing of two people in a singlephotograph. The customer images in a product order can be analyzed usingthe aforementioned methods. An image with a highest score can be chosento fulfill the product intent. If no images score above a definedthreshold, the algorithm can be designed so that a different recipe isautomatically chosen and the process repeats itself with the new recipe.

Often, when doing image splitting, it is necessary to know not onlywhere the main subject is, but also to know the detail surrounding themain subject. For example, the artist may define and store a reciperequiring that a template with three window openings have a main subjectface in a larger left window and low color non-face and non-main subjectin the two smaller windows to the right. Image segmentation algorithmsprovided on the computer system along with people or main subjectdetection algorithms are oftentimes adequate to fulfill aesthetic imagesplitting requirements depending upon the artist's template recipe. Amore advanced splitting technique uses seam carving as described by A.Shai, S. Ariel in “Seam Carving for Content-Aware Image Resizing”, ACMTransactions on Graphics, Vol. 26, Issue 3, July 2007, to determine lowenergy regions of the image. This document is incorporated by referenceherein in its entirety. These low energy regions are ideal candidatesfor splitting and can significantly aid the creative designer to definetemplate recipes for attractive coordinated image displays.

Stored template recipe files created in advance by an artist can applyto the overall image enhanceable item 300, as well as for each windowopening within the item. Each window recipe file can be combined andstored with other window recipe files to generate a stored templaterecipe file. These recipes can be designed and executed on the computersystem for determining parameters such as, but not limited to, landscapevs. portrait orientation, colorfulness, sepia, color tint, person, face,main subject, size of person/face/main subject, location ofperson/face/main subject, surrounding regions of person/face/mainsubject.

In a preferred embodiment of the present invention, in order to sizeobjects correctly, a relative measurement algorithm is used. Thisprogrammed system can specify size and location of an object in an imagevia a variable measure called a unit. A unit can be defined as the widthor height of a face, the width or height of a main subject, or the widthor height of an object in the image. A window recipe file can specifyrequirements to fill a window with a single person, that a face is 20%fill; where face location is specified as two unit spacings from a leftedge, one unit spacing from a top edge; where there is low texture inthe area of the image including two units of spacing to the right of theface or main subject; where the colorfulness of the person'sshirt/blouse/dress (at 2 units below face) is high. The capability tospecify and store in the computer system these required image attributesas window recipes for later repeatable use provides extensive creativeflexibility for an artist.

In another preferred embodiment of the present invention, images can bescored on a merit system. The image that scores highest for a particularwindow recipe is indicated as assigned to the corresponding window. Thiscan be programmed to occur automatically upon activating an applicationon the computer system, provided that there is a number of computeraccessible images available, with no user interaction. This frees theuser from the laborious and time consuming task of looking through manyimage files and is able to automatically create an image enhanceableitem 300 that is often much superior to what a lay user could createbecause the designed look was created by a skilled artist. Of course, ifthe user is not happy with the automatic selection of templates andimages, he/she can manually override the system, by selecting a usercontrol option provided by the program, and either rescale, replace,re-crop, re-enhance, or modify an image using various off-the-shelfsoftware products for digital image editing, or designate a differentimage from his or her collection of images for a particular virtualtemplate window opening.

An exemplary recipe for a window, which can be one of a plurality ofwindows in an image template, is illustrated by the following codesegment:

<MetaDatum Label=“EastmanKodakCompany” Tag=“Orientation” Datum=“1” /><MetaDatum Label=“ EastmanKodakCompany ” Tag=“OrientationWeight”Datum=“20” /> <MetaDatum Label=“ EastmanKodakCompany ”Tag=“FacialCloseToNum” Datum=“3” /> <MetaDatum Label=“EastmanKodakCompany ” Tag=“ FacialCloseToNumWeight” Datum=“10” /><MetaDatum Label=“ EastmanKodakCompany ” Tag=“Favorite” Datum=“1” /><MetaDatum Label=“ EastmanKodakCompany ” Tag=“FavoriteWeight” Datum=“10”/> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“Resolution”Datum=“2431800” /> <MetaDatum Label=“ EastmanKodakCompany ”Tag=“ResolutionWeight” Datum=“100” /> <MetaDatum Label=“EastmanKodakCompany ” Tag=“OffsetLeft” Datum=“1” />

In the above recipe, each required image attribute entry is delineatedby brackets and, where appropriate, is followed by an entry defining itsweighting factor. In the first line, Orientation is the image requiredattribute, and the value paired up with Orientation key is the value 1,which in this example represents landscape orientation (“0” indicates aportrait orientation). Thus, landscape images will receive an attributescore of 1, portrait images a score of 0. For this example and theothers that follow below, the ordering of the entries is arbitrary. Theweighting factor is used for scoring images as is explained below.

The FacialCloseToNum required image attribute indicates that the inputimage should contain close to three faces according to its value. Animage with exactly three faces would receive a score of 1. Images with 2or 4 images receive a score of 0.5, images with 1 or 5 images receive ascore of 0.25, images with zero faces receive a score of 0, images with6 images receive a score of 0.125, images with 7 or more images receivea score that can be calculated byscore=1/(2^(ActualFaces−DesiredFaces)), where DesiredFaces is the Datumvalue of 3 in the FacialCloseToNum attribute line. Similar attributetags can be programmed, using any range of assigned values or byapplying various value formulas, for FacialNumExact (an exact number offaces in an image), FacialGreaterThanEqualToNum (for a minimum number offaces in an image), and FacialLessThanEqualToNum (for a maximum numberof faces in an image).

The Favorite tag defines whether the required image attribute indicatesusing one or more of a user's designated favorite image or images.Favorite images will get a value of 1, non-favorite images will get avalue of 0. Of course, other value assignment strategies can beprogrammed into any window or template recipe.

The resolution tag defines a resolution based on pixels required for theimage in the corresponding window. In this case, the image is requiredto have an image attribute resolution of 2431800 pixels (computed as(width in pixels)×(height in pixels)) or higher to obtain the resolutionweight added to its overall score. Images with total number ofpixels >=the Datum attribute receive a value of 1, all other imagesreceive a value of 0.

The weighting factor as used for required image attributes enables theartist to specify a different level of importance for each requiredimage attribute when determining the image's overall score. Theweighting factor multiplied by the image attribute score generates aweighted score for each attribute. The sum of all weighted scores givesa total score for each image evaluated by this computation method.Clearly, various procedures for determining compatibility of images withrequired attributes can be devised and easily programmed for use withthis embodiment of the invention. The image with the highest total scoreis the image that most satisfies, or is most compatible, with the windowrecipe. In a preferred embodiment of the present invention, if a weightline is omitted for a particular attribute, the default weight of 10 isautomatically assigned. Attributes which are more important have weightsgreater than 10, and attributes which are less important have weightsless than 10. If an attribute is mandatory, a weight of 100 (or higher)is assigned. As used herein to modify the term “attribute(s)”, the terms“required” and “requirements”, etc., are variable requirements accordingto the weights ascribed to any particular attribute. Thus, an attributerises to the level of “mandatory” if its weight is programmed with avalue of 100. For example, in the above code segment, theResolutionWeight is equal to 100. This means that only images with2431800 pixels or higher can be used in this window opening.

All entries, except the last one in the above example recipe segment,are used for scoring candidate images. The last entry, in this exampleit's “OffsetLeft”, is used as a template/window specification, and notas an image selection guideline, and thus the last line does not have apaired weight line. The last entry specifies that the highest scoringimage, or the image selected for a virtual template opening, should bemodified in some respect. This can be referred to as a “post processing”step because it is used as a fine tuning step or as a post-selectionlayout step. In this example the specification states that the mainsubject be shifted left using the rule of thirds before virtualplacement into the template opening. The rule of thirds is aphotographic composition rule that has proven to yield well balanced ornatural looking prints by the average observer, and can be programmedfor use by computer system 26. If an image is broken into an equallyspaced 3×3 grid with two horizontal and two vertical lines, theaesthetic design idea is to move or place the image such that thesubject of interest is located on one of the four dividing lines.Similar rule of thirds tag entries include “OffsetRight”, “OffsetTop”,“OffsetBottom”, “OffsetUL”, “OffsetUR”, “OffsetLL”, and “OffsetLR” toplace (or move) the image such that the main subject is located on oneof the four rule of thirds lines or, as in the last four examples above,on one of the four intersecting points of these lines. Other templatespecification tag entries, for post-selection layout, exist to modify aspecific main subject/face size and location, or overall imagerendition, such as “sepia”, “B/W”, “colorful”, “light”, “high Contrast”,“soft”, “cartoonization”, “emboss”, and other artistic or aestheticeffects.

In another preferred embodiment of the present invention, favorite orprimary images can be designated as such in an images recorded metadataand used as either the focal point of the image enhanceable product, canbe tagged for special or unique treatment either at pre- orpos-selection, or can be used as a primary template image such thatimages in other template windows must conform, as defined in theirassociated required image attributes, in some aspect to keycharacteristics of one or more image attributes of favorite or primaryimages. For example, a large window opening in the center of a templatemight be appropriate to place one of the customer's primary images. Suchan opening can be designated as a primary opening in a template and itsrequired image attributes can be defined such that it will receive animage designated as primary or favorite. A user of the computer systemcan be programmably prompted to identify such images during operation ofcomputer system 26 or such images might have a “primary” indicationalready stored in association with the image as a recorded metadata tag.Other template openings can also be designated as primary, or assecondary, etc., and image files can include metadata informationindicating which ones are favorite or primary images so that theprogrammed computer system can search and find these images forplacement into the designated template window. As mentioned earlier, acustomer can designate an image as a primary or favorite image, candictate the location of their favorite image within one of a number oftemplate windows, or may tag many images as favorites, letting theprogram automatically pick from a selection of favorite images usingother image attributes as selection criteria based on required imageattributes defined in template or window recipes. Other images inneighboring template openings can then be automatically selected bycomputer programming according to window recipes stored in associationtherewith. Other features of window recipes can include detected similarcolor and tone characteristics, or other stored image aspects, of theparticular favorite image or images.

For the example recipe defined by the program segment above, an imagewould be required to have the following image attributes: it would haveto be designated as a favorite image, be laid out in landscape mode, bea group shot, optimally with 3 people in it, and needs to have aresolution of 2431800 pixels or higher to obtain the perfect total scoreof 40 in this example (the resolution weight of “100” is not an actualweight because the number “100” triggers a special case handling in theprogram that the resolution weight is mandatory, and not just a relativelevel of a requirement). The total score is obtained by evaluating eachattribute independently, then summing together each image's weightedvalues. Also in this recipe example, as dictated by the attributeweights, aspect ratio, with a weight of 20 is more important to theartist than most of the other scoring attributes which each have aweight of 10.

The above required image attribute values and weights (in XML these arereferred to as key value pairs) are designed by the artist and arestored in computer storage or memory 40 as definition files for eachtemplate opening (i.e. “window recipe”) as a method to realize theartist's aesthetic intent. In addition to defining attributes on awindow by window basis, openings can be referenced by a group name, ortemplate name, where all window openings in a named group, or template,have at least one common attribute (i.e. “template recipe”). Inaddition, a grouping of all openings in a template can be programmed forenabling global required image attributes, where such attributes arerequired for all openings in the template. A precedence procedure canalso be programmably defined, for example: required image attributes fora group of openings supersede global required image attributes, andindividual opening required image attributes supersede group attributes.For example, the artist may define a global required image attributerequiring that all openings must contain people by specifying under theglobal recipe:

<MetaDatum Label=“ EastmanKodakCompany ”Tag=“FacialGreaterThanEqualToNum” Datum=“1” /> <MetaDatum Label=“EastmanKodakCompany ” Tag=“FacialGreaterThanEqualToNumWeight”Datum=“100” />

By weighting such an attribute with a value of 100, it becomesmandatory. Therefore, if there are no satisfactory images found afterperforming an automatic programmed search through available image databases the program may notify the user that it can't be fulfilled. In theabove global attribute example, if a weight were not explicitlyspecified, a default weight of 10 is automatically assigned in apreferred embodiment. Subsequent examples herein will not specify weightvalues for purposes of simplicity of description.

In another preferred embodiment, a window recipe might define requiredimage attributes such that each window includes an image having aparticular time-date stamp, GPS location, camera parameters, or otheruser definable tags. For example, a window recipe can include anycombination of the following example required image attributes:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“DateMin”Datum=“10/20/2008” /> <MetaDatum Label=“ EastmanKodakCompany ”Tag=“TimeMax” Datum=“21:00” /> <MetaDatum Label=“ EastmanKodakCompany ”Tag=“Date” Datum=“10/20/2008” /> <MetaDatum Label=“ EastmanKodakCompany” Tag=“GPS” Datum=“(44.613934, −110.558167)” /> <MetaDatum Label=“EastmanKodakCompany ” Tag=“GPSRadius” Datum=“75.5m” /> <MetaDatumLabel=“ EastmanKodakCompany ” Tag=“ISO” Datum=“400” /> <MetaDatumLabel=“ EastmanKodakCompany ” Tag=“Title” Datum=“Disney” />

The first two examples above define the minimum and maximum date andtime of capture of the image. DateMax and TimeMin similarly exist. Inthe above example for DateMin, the full month/day/year is specified.This format can be changed to any of a number of accepted formats likeday/month/year. It is also acceptable to include partial dates. If theday of the month is to be omitted, the above Datum would be “10/2008”.The third example shows an example of specifying an exact date. Asimilar field exists for Time. In addition to specifying an exact dateas above, well known holidays can be substituted, such asDatum=“Thanksgiving”, or Datum=“Memorial Day”. Similar substitutions of“Noon” and “Midnight” can be used for the time entry. The fourth andfifth examples require the image to be captured at a particular GPSlocation, such as a campground near a particular tourist attraction(within 75.5 meters in this example) in Yellowstone National Park; thefifth example allows control over the camera's image capture conditionssuch as ISO, and can be used to specify more advanced features such asfocus distance, F#, etc. The final example can be used to require imageswith particular customer or third party provided metadata tags. In thisexample, recorded metadata associated with an image is required toinclude the text “Disney”. For another example, it is possible for auser to specify for a window recipe only shots of “Uncle Joe”, or for aKiosk application to require looking for images on a particular websitethat are metadata tagged for specific usage on a particular template ora particular family of templates. For example, the program can bedesigned to look for images tagged with “Uncle Joe” on the user's orusers friend's Facebook account, and such an account on a computersystem accessible web server can automatically be searched and imageshaving the required metadata retrieved; or images from Flickr that aretagged as “sunset” can be required in a window recipe; or customdatabases can be created locally on the computer system or on a coupledcomputer system in a distributed network, or on network accessibleservers with the intention of combining images stored in them into userenhanceable products.

Public or private image databases may have tagged images with specificcontent available for usage in the user enhanceable product 300. Thiscontent may be generic such as “beach”, “palm trees”, or may have aunique identifier with a product label such as “Kodak Premium”, or maybe authorized by and associated with a particular celebrity such as a“Famous Designer”, or may be part of a special store promotion such as“Walmart Summer 2008”. More than one Title recipe entry can exist to taga window opening, for example “Famous Designer Fall 2008 Collection” or“Colorful Leaves” is required to appear together with “Disney”. Theusage of such template recipes can be used to entice consumers topurchase the latest celebrity branded and celebrity approved imageenhanceable item 300 and may include the automatic diversion ofroyalties to said entity and entice the return of customers time andtime again to purchase the latest offering by a specificcelebrity/artist/sports/famous person or event such as a World Series2009 collection.

Once again, the programming of global recipes and local recipes for eachwindow can be used to generate recipes for templates of various types.For example, when creating a picture book, it is often desirable, butnot mandatory, to arrange pictures in chronological order. We can createglobal recipe for a picture book template to “softly” enforce such rulesfor each window in the virtual picture book template, as shown below, byassigning a lower weight values for “soft” enforcement:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“Date” Datum=“W1>=W0” />

In the above example, the recipe requires that W1 (window 1) should havea date (or time stamp) that is greater than or equal to W0 (window 0), agreater time stamp meaning later in time, where W1 and W0 are moregenerally described as W<integer>, where integer is any window openingID. By modifying the assigned weight of the above recipe, we can enforcemore or less the desire to maintain a chronological ordering of customerimages in a picture book as compared to other image attributes. To forcechronological ordering as a mandatory entry, the weights would be set to100 for this entry for each window opening.

In another preferred embodiment of the present invention, the windowrecipe might include color intent such that each window opening in atemplate has a defined stored color intent. For example, a recipe cancontrol color content as follows:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“ColorIntent” Datum=“Cool”/> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“ColorIntent”Datum=“Cyan” /> <MetaDatum Label=“ EastmanKodakCompany ”Tag=“ColorIntent” Datum=“150,220,75” /> <MetaDatum Label=“EastmanKodakCompany ” Tag=“ColorIntent” Datum=“W0” />

The first entry defines the required color attribute via values such as“warm” and “cool” for the image; the second entry defines the allowablecolor attributes via specific color values as “red”, “green”, “blue”;the third example defines the allowable color attributes via specific24-bit RGB code values to define required image attributes with aspecific color value to match, for example, a company logo, pantonecolor, or other pre-defined color scheme. The final line is the mostcomplex: if the attribute value (Datum) is a W<integer>, this indicatesthat the primary color from an image disposed in window opening<integer> is first to be extracted, and then the current window openingis defined to require image attributes to have primary colors similarto, or dependent upon, that of the image in window <integer>.

In the previous examples, images that have attributes as defined in therequired image attributes score higher than those that do not. After thebest image is selected, the image is further modified to accommodate thewindow recipe command. So, for the last example, if W0 had a warmprimary color, the image that matches this warm color the closest getsthe highest weight. Upon final selection of an image, the image selectedfor this window opening is further checked and modified, i.e. colors aremodified to warm properties, if necessary, to obtain this same warmprimary color from window W0.

In another preferred embodiment of the present invention, the windowrecipe might include tonescale intent such that images in each windowopening in a template has a predetermined tonescale. For example, eachrecipe entry line can control tonescale in any of the following ways:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“ToneIntent” Datum=“Light”/> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“ToneIntent” Datum=“Low”/> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“ToneIntent” Datum=“W0”/> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“ToneIntent” Datum=“~W0”/> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“ToneIntent”Datum=“W0|W1” /> <MetaDatum Label=“ EastmanKodakCompany ”Tag=“ToneIntent” Datum=“~(W0|W1)” />

The first example controls the allowable image scene contrast attributesvia values such as “light” and “dark”; the second example controls theallowable image contrast intent via specific attribute values such as‘high, “low”, “normal”. The next example is similar to the color intent.If the required image attribute value is defined as W<integer>, thisindicates that the primary tonescale characteristics from an image inwindow opening <integer> is first to be extracted, and then the requiredimage attributes for an image in a current window is chosen to have atonescale similar to, or dependent upon, that of an image in window<integer>. This feature can be used in conjunction with or independentlyfrom the designation of a favorite image, where the color/toneinformation is extracted from the favorite image and used in otherwindow openings to define required image attributes. This is a powerfulprogram extension, for example, that would allow, for example, the colorand tone of all images in a template to match a certain reference image.The certain reference image might or might not be included in thetemplate. The reference image might be of a paint swatch, fabric, or animage of a room or any other item.

This recipe type can also be used in other commercial applications. Forexample, a window recipe can be created for a school yearbook wherebyeach graduate picture is a window having defined required imageattributes associated therewith, and the color and tone of each picturebe made to match, or be compatible with, a reference image. Similarapplications exist for advertising. A mail order catalog or weekly storecircular can use window recipes to describe page layouts such that allor a group of images have similar contrast. A programmable storefrontdisplay can be advantaged with similar required image attribute designs.

In the third example, if the required attribute value is ˜W<integer>,this indicates that the primary color/tone characteristics from an imagein window opening <integer> is first to be extracted, and then thecurrent window opening is chosen to include an image having a color/toneopposite to that of the image in window <integer>. So, if the window<integer> image is high contrast, the current window opening will havean image with low contrast. Accordingly, the higher the contrast inwindow <integer> image, the lower the contrast in the current windowimage.

If the attribute value is (W<integer1>|W<integer2>| . . . |W<integern>),this indicates that the primary color/tone characteristics from imagesin windows <integer1> . . . < integern> are first to be extracted, andthen the required image characteristics in the current window opening ischosen to have a color/tone similar to the average of the images in then window openings. Similarly, if attribute tag value is ˜(W<integer1>|W<integer2>| . . . |W<integern>), this indicates that the primarycolor/tone characteristics from images in window openings <integer1> . .. <integern> are first to be extracted, and then the current windowopening is chosen to have an image with color/tone opposite to theaverage of the n window openings.

Using the color and tonescale required image attribute values, theartist can design all window openings to have similar color/tonecontent, unique color/tone content, or some combination thereof. Itshould be readily understandable that these two examples can be modifiedto include other required image attributes such as texture,colorfulness, season, time of year, time of day, people, sharpness, andnoise.

In situations where a customer manually adjusts thecolor/tone/sharpness/cropping of an image in a window opening via theuser control option that affects the color/tone/sharpness/cropping ofanother window in the project whose required image attributes aredependent upon such characteristics of the image, the dependent imagecan be programmed to be automatically adjusted accordingly. For example,if window opening W5 is defined to have an image with identical contrastas the image in W1, and the user then manually adjusts the contrast ofthe image in W1, making it higher, the contrast of the image in windowopening W5 will also increase automatically according to the program. Insome situations, the user may choose not to activate this feature, sothe user control option can be configured accordingly.

In another preferred embodiment of the present invention, the windowrecipe might include further object detection or scene reflectionrequired attributes. For example, the window recipe can dictate morespecific requirements for each window opening:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“Object” Datum=“Balloon” /><MetaDatum Label=“ EastmanKodakCompany ” Tag=“Theme” Datum=“Fall” /><MetaDatum Label=“ EastmanKodakCompany ” Tag=“Similar” Datum=“W1” />

The first example controls whether the window opening should have aspecific object in the scene (a balloon), and can also specify otherobjects such as “BirthdayCake”, “SoccerBall”, or “Dog”. The secondexample controls a required attribute defining a theme for the openingwith a value such as “city”, “colorful”, or “dreamy”. The former examplecan be used to search for and select all images from a soccer game or abirthday party. The latter example can be used to define a proper moodfor a particular window opening. The last line utilizes a similaritymeasure between images. It defines the image in the current opening tobe similar to that of Window opening 1. This feature supports the ‘˜’attribute which indicates “dissimilarity” in images, and the ‘|’ featurewhich indicates averaging several windows together. The similaritymeasure can use any number of image attributes, but RGB histogramstatistics such as Chi-squared analysis or low resolution pixelcorrelation metrics can also be used, are fast, and perform quite well.Two low resolution metrics that perform well are Mahalanobis distanceand edge map correlation. The Mahalanobis distance is similar tostandard deviation, but also takes into consideration the correlationbetween each of the individual color channels. The edge map correlationis performed on low resolution images (64×96 pixels or smaller) bycomparing the derivative (using Sobel filter) of the reference image tothe derivative of all other images. More complex similarity metricswhich utilize complex semantic information can be readily incorporated.

In another preferred embodiment of the present invention, the recipe caninclude further object, and scene attributes, such as in the followingexamples:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“Location” Datum=“Beach” /><MetaDatum Label=“ EastmanKodakCompany ” Tag=“SceneContains”Datum=“Snow” />

The top example controls the ability to specify a generic locationrequired attribute value such as “water”, “beach”, “city”, or “indoors”.These locations are overall scene classifications, and the resultingclassification is based on an output from a scene classifyingcomputation carried out on a digital image. The last example can furtherconstrain the image selection procedure by requiring that scenes in animage contain certain objects or characteristics such as “snow”,“water”, “sky”, “foliage”, “dirt”, and “skin”. The latter example can beimplemented via belief maps. If no snow is found in an image, a value of0 is returned as that image's attribute value for “snow”. The more snowfound (the greater the percentage of snow pixels), the higher is itscorresponding value for the required “snow” attribute in the windowrecipe. A discussion of sky detection is provided in Gallagher, et al.,“Detection of Sky in Digital Color Images”, U.S. Pat. No. 7,336,819 andan example of foliage detection is provided in “Color Segmentation as anAid to White Balancing for Digital Still Cameras”, by Cooper,Proceedings of SPIE, v4300, p 164-171, 2001. Both of these referencesare incorporated herein by reference in their entirety.

In another preferred embodiment of the present invention, the recipe caninclude further people attributes, such as in the following examples:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“AgeLessThanEqualtoNum”‘Datum=“2” /> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“FacialPose”Datum=“Left” /> <MetaDatum Label=“ EastmanKodakCompany ”Tag=“FacialEpression” Datum=“Angry” /> <MetaDatum Label=“EastmanKodakCompany ” Tag=“EyesOpen” Datum=“1” />

The first example attribute requires that the age of people depicted inthe image be less than or equal to a certain number, in this exampleless than or equal to two years old. Similar attribute requirementsexist for AgeGreaterThanEqualtoNum, and AgeClosetoNum. Often we wantpeople facing “forward”, but, it is often desirable for a person orpersons pictured in one window to be looking in the direction of asecond window opening. The second example requires a particular facialpose of people or a person depicted in the image, in this example theperson should be looking left. The next example above indicates apreference for a facial expression or demeanor. This allows control overfacial expressions such as “angry”, “sad”, “surprised”, “happy”,“smile”, and “neutral”. The last example can be used to make sure theeyes of all found faces are open. Similar tags can be written to controlrelated features such as ethniticity, gender, hair color/style, eyecolor, skin tone, height, age, facial or hair accessories (sunglasses,headband), facial hair, and mouth open/closed. These controls are wellknown algorithms familiar to persons of skill in the art, and aresimilar to those previously disclosed.

The next example utilizes facial recognition to identify a person in animage.

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“PersonID” Datum=“UncleJoe” /> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“PersonID”Datum“W0” />

In the first example, the required personID attribute controls whichperson's image is being inserted into an opening. The attribute value“Uncle Joe” can be satisfied either by searching for and findingrecorded metadata associated with the image, or by utilizing anaccompanying facial recognition database and extracting the metadata. Itis well known that extracted metadata can thereafter be recorded withthe image and need not be obtained by repeating the extractionprocessing later. Many cameras and imaging software packages now includefacial recognition capabilities. The last example above utilizes thewindow relationship feature for window recipes. This last example saysthat the image being placed into the current opening must have a person,and that person must also be in window W0. This last example is ageneric way of describing a relationship of people appearing in morethan one template window. For example, we can have a center templatewindow contain a closeup of a single person as dictated by a favoriteimage being placed therein, and all surrounding template windows maythereby be defined to require that same person somewhere in the image.

In another preferred embodiment of the present invention, the recipe caninclude further image selection steps, such as in the followingexamples:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“AspectRatio” Datum=“1.5”/> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“WindowShape”Datum=“Oval” />

The first example attribute defines the window opening width over heightaspect ratio. In this case, we know the width is 1.5 times the height.When used together with the second example, defining an oval, we candefine a maskable area to overlay on top of any candidate images. Wethen use the maskable area to score each of the candidate images.WindowShapes can include Rectangle, Circles, Ovals, Pentagrams,Hexagons, RoundedCornerRectangles, and can be easily extended to anyother shape. When AspectRatio and WindowShape are used in combinationwith other recipe items such as PersonID, FacesClose2Num, etc, only themaskable area is included in the overall score. For example, a 1.5aspect ratio rectangle may include 5 faces, while a 1.5 aspect ratiooval may only include 3 faces and would result in different scores forthe same underlying image.

In another preferred embodiment of the present invention, the recipe caninclude multiple image selections per window opening. This is defined ashaving two or more recipe entries for each window. In the case ofoverlap, the last recipe entry is the top entry unless the Order entryis used. For example, we may have the following two examples for asingle window opening:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“AspectRatio” Datum=“0.5”/> <MetaDatum Label =“ EastmanKodakCompany ” Tag=“WindowShape”Datum=“Oval” /> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“PersonID”Datum=“Uncle Joe” /> <MetaDatum Label=“ EastmanKodakCompany ”Tag=“PersonID” Datum=“Aunt Suzie” /> <MetaDatum Label=“EastmanKodakCompany ” Tag=“FacialEqualToNum” Datum=“2” />Along with the following second recipe for the same window opening:

<MetaDatum Label =“ EastmanKodakCompany ” Tag=“AspectRatio” Datum=“0.5”/> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“WindowShape”Datum=“InverseOval” /> <MetaDatum Label=“ EastmanKodakCompany ”Tag=“FacialEqualToNum” Datum=“0” /> <MetaDatum Label=“EastmanKodakCompany ” Tag=“Location” Datum=“Sand” />

The first recipe specifies an oval mask, half as wide as tall. We wantexactly two people, Uncle Joe and Aunt Suzie. The second recipe alsospecifies an oval mask, half as wide as tall, but, it is an inversemask. So, the image area outside of the oval is used for image scoring.In this case, the second image outer mask area image should have nopeople and contain sand.

The above two examples assume the image is centered and fills the windowopening as much as possible. It is possible to do various collage typeentries for each opening, by specifying the subwindow [X,Y] offset andthe subwindow width and height for each of the recipes that fulfill aparticular opening.

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“SubWindowX” Datum=“0” /><MetaDatum Label=“ EastmanKodakCompany ” Tag=“ SubWindowY” Datum=“0.5”/> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“SubWindowW” Datum=“1”/> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“SubWindowH” Datum=“0.5”/>

In the above example, the image that scores highest for this particularrecipe will be fit to the lower half of the window opening. Values forSubWindowX, SubWindowY, SubWindowW, and SubWindowH scale between 0 to1.0, where (0,0) is the upper left corner and (1,1) is the lower rightcorner, but other conventions can be used. Finally, these subwindows canbe feathered into one another and order can be assigned as follows:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“Feather” Datum=“0.1” /><MetaDatum Label=“ EastmanKodakCompany ” Tag=“ Order” Datum=“0.75” />

In the above example, feather command controls the feathered edges on animage. The convention used here is that the feather value can be anypercentage of the window opening width. The Order value assigned afloating point hierarchy to each image in a window opening, where highervalues move the window further to the front. If no Order value isspecifically stated, or if we have two identical Order entries, it isassumed the order of the current recipe is in front of the previouslyfound recipe for the same window opening. If the first recipe per windowopening does not have an Order value, a default of 0 is used, whichmeans it will be in the back, and all other images will be placed infront of it. Using masks, inverse, masks, locations, and order enable apowerful scripting language for the skilled artist.

In another preferred embodiment of the present invention, the recipe caninclude the demarcation of specific regions of the image. In particular,region segmentation of sky, snow, skin, foliage is often determined bycolor and location within the image. Background areas are often denotedby lack of texture. These regions can automatically be masked off andcombined or replaced with other attribute areas:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“SelectSkyInverse”Datum=“0” /> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“SelectFace”Datum=“50” /> <MetaDatum Label=“ EastmanKodakCompany ”Tag=“BackgroundColor” Datum=“W0” />

The first example can be used to set the transparency of the non-skyregion of the selected image to 0%, or set the transparency of the skyto 100%. The Datum value indicates the transparency value. When thisimage is overlaid on top of a previously fulfilled recipe for the samewindow opening, the sky will be replaced with the image contents of thebackground image. When the Select tags are used, all areas not includedare set to a transparency of 100, such that they will not be seen in thereproduced image. In the second example, we are selecting just thefacial region of an image, the rest of the image will be transparent.The face will have a 50% transparency, such that it will be blended withthe existing background image at 50%. When used with SelectFaceInverseof a second image, we can swap the face of one person with another. Theresults of the face swap are much better if the exact facial outline isselected using one of the previously mentioned examples. Finally, in thelast example, the background of an image is selected based upon imageenergy, where energy is described by Avidan and Shamir in “Seam Carvingfor Content-Aware Image Resizing” in ACM, Vol. 26, No. 3, July 2007,incorporated by reference herein in its entirety. The low energy area isconsidered background area, and in this case, the color of thebackground area will be made to match the primary color of what was inwindow opening 0.

The first example can be used to create adaptive montages of eitheraesthetic or practical significance. Aesthetically, special effects canbe created, for example, in an advertising poster to add an image ortext in a sky or background area of an image. The second example canalso be used to create artistic effects, but can also be used to replacefaces that have inappropriate facial expressions, eye closers, headpose, etc, with faces that are more appropriate. The second example canalso be used in commercial purposes to replace, for example all theheads of models in the store circular with the children in ones family.A custom softcopy circular can be created for each household, not onlywith custom targeted content, but with family members wearing theclothing, operating equipment, or looking on in the distance. Socialwebsites like Facebook can use such recipes to create custom slideshowsfor both pleasure and profit. The latter example can be used in contextwith the previously described adaptive recipes such that the end productmore closely follows a theme or more closely resembles the artistsintent. For example, the background information can be made to have thetexture or color of a favorite image, made transparent or just driven towhite so that the window opening can concentrate on the main subject.

In another preferred embodiment of the present invention, the recipe caninclude the usage of units in window recipes to specify the size andlocation of a particular face, main subject, or other object. Forexample:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“FacialEqualToNum”Datum=“1” /> <MetaDatum Label=“ EastmanKodakCompany ”Tag=“FacialEqualToNumWeight” Datum=“100” /> <MetaDatum Label=“EastmanKodakCompany ” Tag=“FaceSize” Datum=“0.5” /> <MetaDatum Label=“EastmanKodakCompany ” Tag=“FaceLocation” Datum=“Center” />

The first recipe line defines the current window required imageattribute needs an image with one face. The second line says thisrequired image attribute is mandatory—it is mandatory for the image tohave exactly one face. Images containing 0 or greater than 1 face arenot considered for inclusion in the current window opening. If no imageswith exactly one face are found, the recipe cannot be fulfilled, and (aswith all other unfulfilled mandatory required image attributes) the jobcan be terminated, a notification can be provided to the user, an optionto override the requirement can be provided, or an option for the userto insert a selected image can be provided, or the user can be promptedfor some other type of intervention. The third line defines that thisface must occupy 50% of the image area. Images with slightly smaller orlarger faces will score high, images with large face differences willscore lower. One formula used is 1−abs(TargetFaceSize−ActualFaceSize),but non-linear variants such as logarithmic, gamma, and similarfunctions can also be incorporated. Regardless of the preferredvariations for evaluation, an image with one face, covering closest to50% of the window area will be calculated to score the highest and beselected for inclusion in the template window. After selection, theimage can be specified for post-processing and be resampled such thatthe face will occupy exactly 50% of the window area. The final lineabove specifies to center the face in the middle of the window opening.The FaceSize and FaceLocation key value pairs include the W0 and similaroptions such that the size and location of faces can mimic that of thefavorite images placed in the template. In addition, because sizing isso critical, the FaceSize and FaceLocation key value pairs include themore advanced W0 notation, such as:

Tag=“FaceSize” Datum=“W0*0.8” /> <MetaDatum Label=“ EastmanKodakCompany”

This defines, for a window recipe, that the face in the current windowopening should be 80% of the size of the face in window opening W0. Whenmultiple faces occur, the FaceSize and Location refer to the averageface size and centroid of all faces. When W0|W1|W2 notation is used, itdefines using an average size face from window openings W0, W1, and W2.In addition to “Center”, FaceLocation can include the rule of thirdlocations, UpperLeft, UpperRight, LowerLeft, LowerRight, as well asUpper, Lower, Left, Right. The W0 notation for FaceLocation includesmirroring the location such as:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“FaceLocation”Datum=“W0*HorizMirror” />

The above recipe line will mimic the horizontal location of the face—soif the face in W0 was in the upper third, the face in the current windowwould occur in the lower third of the image. To make relative locationsas compared to W0, offsets are used such as:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“FaceLocation”Datum=“W0+0.2H−0.3V” />

Where the above line says the face location in the current window shouldbe the same as the face in window W0, but, should be offset 20% to theright and 30% towards the bottom. The above commands are illustrated foruse in detected faces, but equivalent commands exist for main subjectsize and location as well as object size and location when the mainsubject is something besides a face.

The process of evaluating the fitness, or compatibility, of multipleimages as against a window recipe for a particular window is shown inFIGS. 4-6. FIG. 4 shows a window recipe for a window in a userenhanceable product, such as a picture frame. The required imageattributes include: a favorite image with weight 20; contain 2 faceswith weight 40; contain frontal pose with weight 20; and be indoors. Thelast two lines in FIG. 4 are used as a template/window specification forpost selection layout, and not as an image selection guideline, and thusthe last two lines do not have a paired weight line. Optionally, a userwould then select their favorite image or images to be placed inprominent windows or other locations in the virtual template. FIG. 6illustrates an example group of searchable images. There are n imagesdisplayed on the left. Each image's fitness will be evaluated accordingto the window recipe in FIG. 4. For the sake of simplicity, we willassume that the user has preselected image2, 620 and imagen, 640 astheir favorites.

FIG. 5 shows the method of evaluation computation of each required imageattribute for each image attribute. For each image attribute, we have avalue, which is determined either by a metadata entry or semanticalgorithm. This value is between 0 and 1.0, where 0 is the lowestpossible score, and 1.0 is the highest possible score. After multiplyingby the accompanying attribute weight, a resulting weighted value foreach required attribute per image is obtained. After summing up allweighted values for each image, a total image score is obtained. Theimage with the highest total score is the closest compatibility match tothe recipe.

The first listed attribute is for favorite images. 620 and 640 receive avalue of 1. 610 and 630 receive a value of 0. Multiplying by thefavorite weight of 20, we get weighted values of 0, 20, 0, 20,respectively, for each of the four images shown.

The FacialCloseToNum attribute favors images with 2 faces.Unfortunately, none of the images have two faces—each image having onetoo many, or one too few faces, giving a value of 0.5 for each image.Since the FacialCloseToNum weight was 40, multiplying 40 and 0.5 resultsin a weighted value of 20 for all four images.

The FacialPose attribute favors all frontal poses. Facial poseestimators return the pitch and yaw of a face in degrees. The value iscalculated by setting value=(100−(Pitch+Yaw))/100, and then clippingsuch that the value calculation is normalized from 0 to 1.0. Thisstrongly favors frontal poses where pitch=yaw=0, and penalizes posesotherwise.

The Location attribute is looking for indoor scenes. State of the artindoor/outdoor locators use metadata tags of ISO, exposure time, andflash fire, along with key semantic information such as foliage and skycolored pixels, along with overall color temperature of the scene. Thereturned value is an estimate that the scene is indeed an outdoor scene.For scenes 1, 2, 3, n, the value returned was 1, 0.7, 0.25, and 1,yielding weighted values of 9, 7, 2.5, and 10.

With the evaluation of the above weighted values for each image andattribute completed, we sum the weighted values for each image, giving atotal of 49, 67, 42.5, and 55 for images 1, 2, 3, and n and assign to615, 625, 635, and 645 in FIG. 6. 67 is the highest total score,corresponding to image 620. Image 620 satisfies the window recipeillustrated in FIG. 4 the best and is selected 650 for inclusion in thewindow opening.

The last two lines of the recipe in FIG. 4 provide post-processinginstructions for image 650. The Offset UR says the face in the imageshould be located in the upper right rule of thirds intersection. TheFaceFill attribute indicates that the size of the face should be 20% ofthe width of the window opening. The above two rules are shownschematically, 660 in FIG. 6, along with the final image, 670.

Additional post-processing instructions can be utilized to change thecolor, tone, sharpness of images and change the relative color, tone,contrast to other window openings as follows:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“AutoEnhance” Datum=“1” /><MetaDatum Label=“ EastmanKodakCompany ” Tag=“Contrast” Datum=“<W0” />

Where the first example defines applying an autofix or automatic imageenhancement to the image in the current window. Similar entries existfor AutoNeutralBalance, AutoColorBalance, Auto Contrast, AutoSharpness,and AutoZoomCrop. If AutoZoomCrop is used, and the window opening stillscores highly if small, outlier faces are automatically cropped out ofthe image, they will be cropped out as long as the resulting imageresolution is sufficient for the final enhanceable product. The secondexample indicates that the contrast of the image in the current windowshould be less than the contrast than the image in window 0. Similarentries exist for Colorfulness, Brightness, and Sharpness. In addition,the Datum values of “>W0” would mean the criteria in question would begreater than window 0; and “>>W0” would indicate the criteria inquestion would be much greater than window 0.

The concept of using the > and <characters in the recipe also allowrelative specification of window content. For example:

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“AgeClosetoNum”‘Datum=“>W0” /> <MetaDatum Label=“ EastmanKodakCompany ”Tag=“AgeClosetoNum”‘ Datum=“>(W0+10)” /> <MetaDatum Label=“EastmanKodakCompany ” Tag=“AgeClosetoNum”‘ Datum=“<(W0+30)” />

Where the first example defines having the current window contain aperson whose age is greater than the estimated age of the person inwindow 0. If >> was used, the age would have to be much greater. Theserelative relations can be easily defined within the program as desired,for example, “>” could be defined as five years, and “>>” could bedefined as fifteen or twenty years. The next two examples show how tospecify the age of the person in the current window to be between 10 and30 years the estimated age of the person in window 0. The additionof >, >>, <, and <<along with the offsets, | (averaging), and ˜(inverting) enable a rich suite of tools for the artist.

The described embodiments of the present invention are not limited tothe values and attributes defined herein, and can be extended to otherparameters as an artist sees fit. As computer and hand held devicealgorithms evolve, future available image metadata can be incorporated.While some metadata tags are easy to evaluate, such as time stamp,others are more complex, such as selecting images with balloons. As therecipe is loaded into the program, a list of necessary attributes isaccumulated, and only those attributes that are required for aparticular image enhanceable item 300 are resolved for each image.

Another preferred embodiment of the present invention is the usage ofadaptive recipes. For example, a user is making over a room in his homeand would like to hang decorative image enhanced items on the wall. Theuser can bring a paint sample from the room to a retail kiosk that isrunning a computer system program embodiment of the present inventionsuch as on computer system 26, and the paint sample can be scanned bythe kiosk's flatbed scanner 38. This paint sample can be used to defineoverall color and texture required image attributes for the decorativeimage enhanceable wall hanging. Similarly, a frame or matte can bescanned for defining attributes such as color and texture. If a flatbedscanner is not available to kiosk computer system 26, a calibratedwebcam or video camera at a kiosk will suffice. Similar techniques canbe used if a user brings in a swatch of material, such as a pillow,cushion, curtain, or any other object in the room to be decorated. Notonly can the resultant user enhanceable product be made to match thismaterial, the background matte of the template can be a replication ofthe actual material or paint sample. Finally, a user can take photos ofobjects or walls, and those objects can be used as the reference digitalimage. For example, a user may provide four reference materials, R0 canbe a paint swatch, R1 can be a curtain, R2 can be a pillow, and R3 canbe a sample from a magazine. The following window recipe examples can beapplied to these samples.

<MetaDatum Label=“ EastmanKodakCompany ” Tag=“ColorIntent” Datum=“R0” /><MetaDatum Label=“ EastmanKodakCompany ” Tag=“ToneIntent” Datum=“R1|R2”/> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“MattColor” Datum=“R0”/> <MetaDatum Label=“ EastmanKodakCompany ” Tag=“MattTexture” Datum=“R3”/>

In the first example, the color of an image in the current window can bemade to match the color of R0, which is the paint swatch. This is doneby first digitally balancing the image to neutral and then adding acolor cast to the image—the color cast being a match to the color castof the paint swatch. In the second example, the tonescale of the imagecan be similarly manipulated to match an average contrast of the curtainand pillow. In the third example, the matte color will be adjusted tomatch that of the paint sample. In the final example, the matte texturewill be adjusted to match the texture sample from the magazine clipping.The above use of reference images is very similar to the window matchingentries described above, say Datum=“W0”, but the key difference is thatthe reference image in these examples is not displayed in the completedcoordinated image template design or in the final image enhancedproduct.

Another preferred embodiment of the present invention involves theevaluation of multiple window recipes on a single set of images. Forexample, a Famous Designer may come out with the Fall 2008 line oftemplate recipes. Depending on the images provided by a user, oneparticular template recipe might be more appropriate than another. Forexample, if a customer loads a group of images containing sports scenes,a Famous Designer sport theme template recipe may be used initially tobegin designing the final image enhanceable product. If the customerloads images of children, a Famous Designer infant or toddler theme maybe selected. If a customer loads a group of images containing waterscenes, a Famous Designer beach template may be chosen. The names of thetemplate themes can also be displayed to a user so that the user canselect and open a template initially whose name reflects the type ofcontext that fits the user's image collection. The selection of afavorite image will also cause some templates to score higher thanothers. Based on a different selection of a favorite image, thetemplates that score highest can change drastically. The highest scoringtemplate as calculated by the program can be used, or the top n highestscoring templates can be programmed to be presented to the customer forhim/her to make a final decision as to which is his/her favoritetemplate. The procedure that is followed to identify the types of scenesdepicted in groups of customer images is well known in the art and isnot described further.

If several favorite images are selected by the user, the correspondinghighest scoring templates can be presented to the customer, or the top ntemplates for each favorite image can be presented, or the top ntemplates for only the top scoring favorite image can be presented. Thetypes of such template recipes is virtually unlimited. Image enhanceableitems 300, can include framed templates, clothing items, greeting cards,photo books, and other tangible goods; but also to softcopy goods aswell such as cell phone, PDA, and computer screen savers, digitalgreeting cards, as well as digital picture frames, and automatic digitalpresentations used in store fronts, office lobbies, stadiums, theatricalshows, and for personal enjoyment.

In another preferred embodiment of the present invention, a uniquetemplate recipe is assembled on the fly and pre-selected recipes aredisplayed in a pre-determined or random order. For example, acontinually running slide show on a digital picture frame (which can bea picture frame at home or a large electronic storefront window) mayautomatically extract photos from a public or private website, or from alocal computer. Each and every displayed frame can be unique and can beprogrammed never to be shown again, if so desired. A preferredembodiment of the present invention includes a procedure of writing acontinually looping program that modifies the recipe in each iteration.During each iteration, the top x compositions are displayed in a randomor a predetermined order. When selecting images from online galleriessuch as Facebook or Flicker, or Photobucket, the number of high scoringtemplate configurations can be staggering- and a limit can be imposedfor each iteration of the looping program. The looping program can havenested loops, each changing one or more recipe entries in random orpredetermined order. Fulfilled recipes can be stored in a database toprevent displaying the same recipe twice, or to purposely replay recipesin a preprogrammed or random order.

Template recipe entries can be configured by, and grouped according to,product line and can be upgradeable over time. A similar architecturecan be used in many product lines, with a baseline product only offeringa subset of required image attributes to be selected, a moderate productoffering an extended set of selectable required image attributes, and apremium product offering a full suite of required image attributes. Inaddition, the attribute readers can be designed to be forward compatibleby ignoring any attribute values it does not understand or can notevaluate. This can occur if, for example, a new artist's collection oftemplates uses attributes not recognized by a previous version of theprogram. For example, if balloon object extraction is not supported, itis simply ignored.

Potential designs for interpreting and using the correspondingattributes is numerous. At the highest level, a parameterized andweighted fitting algorithm scores each individual image from a set ofrequired image attributes for each window in the template. The objectiveof the parameterized weighted fitting algorithm is to find the best(most appropriate) image for each window based on image/template imagewindow required attribute comparison. As used throughout herein, a“required” opening attribute (or window attribute) is a soft or hard(depending on weights) requirement, defined in a recipe, that an imagemust satisfy. Image attributes are the metadata describing imagecharacteristics that are extracted by digitally examining an image, orprovided by a user, or generated by a camera, or otherwise provided andstored with an image or generated on the fly. The parameterized weightedfitting algorithm iterates through a list of windows and scores eachimage relative to required image attributes for a window and finds thebest image for that template window.

The algorithm by which a preferred embodiment of the present inventionis implemented is described in reference to FIGS. 7-12. The algorithm isimplemented on a computer system such as depicted in FIGS. 1 and 2 anddescribed above. The user interacts with the computer system via usercontrols 68, 58, also explained above, in response to informationdisplayed on screen 56, 66 as performing actions requested by the useror by prompting the user for information or for making a selection.Referring to FIG. 3, the algorithm starts with presenting a templateselection function 710 on display screen 56, 66. In this algorithmexample and following description, we will assume that the templatespresented for selection are of a form corresponding to a picture frame,as shown in FIG. 3, that the user desires to purchase. Succeeding stepsof the algorithm can be implemented in well known graphical userinterfaces for prompting the user for selecting options or for otherinformation. In a preferred embodiment, the template selection displayedon the screen will physically resemble the actual picture framesavailable to the user for purchase. If a user has selected a mug or atee shirt, then the displayed “virtual” template may be designed toresemble those products so that the user can see a reasonably realisticversion of a desired image enhanced product that includes an imageselected by the user. In the example algorithm illustrated in FIG. 7,the template options are set to default values initially at step 720. Atstep 705 a user selects whether he would like the system to choose atemplate or to participate in the template selection. If the user electsan automatic option then the system will select and store a defaulttemplate at step 770 and then proceed to step 1010. If the user electsto participate then the system will display template selection optionsat step 730. If the user decides at this point, step 715, not to selecta template then the system will revert to an automatic default templateselection. If a user selects a template at step 715 the system willrequest the user to select template characteristics at step 740. If theuser selects all requested characteristics options at step 725 then thetemplate is complete and the next step proceeds to step 810. If the userdoes not provide all options at step 725, default values for unselectedoptions are automatically stored at step 760 and the complete templatespecification is stored at step 780 and proceeds to step 1010.

At step 810 the template was selected by the user and now the imageselection begins. If the user has not yet provided images, either in alocal computer accessible database, a network accessible database or ona portable or other storage device, then at step 820 the system willprompt the use to identify where images can be accessed. The prompt mayor may not occur if a data base has previously been identified to thesystem or if a user has coupled a portable or other image storage deviceto the system. If the user indicates at step 805 that he will selectfavorite or primary images, then at step 830 he will indicate theselected images to the system and at this point the system willautomatically proceed to step 905. If the user indicates that he willnot provide favorite or primary images at step 805, then the system willautomatically select images based on the step as defined in block 1210.At step 840 these automatically selected images will be digitally placedinto the previously selected template. At step 815 the template andselected images will be displayed to the user on a screen or monitor andthe user will have the option to accept the image product, however, thepresent invention includes an option 860 for the user not to review theimage product. If the user accepts or if the user does not review theproduct, the product is output at step 870. If the user does not acceptthe output product at step 815, then the system repeats an automaticimage selection using step 1210, but outputs a template plus images thatis different from any previous outputs. Alternatively, a user controloption 850 can be executed at this point wherein the user performs hisown selection steps instead of the step 1210 performed automatically bythe computer, and the user's selection is then output at 870. The usercontrol option can be implemented via activation of an image editingsoftware that is part of the image enhanceable product software or itcan be implemented by a separate image editing software. The output step870 operates according to the type of image enhanceable product that theuser has selected. For example, and not by way of limitation, if a framehas been selected, then the output can involve a frame with selectedimages being delivered to the user (home delivery or retail outletpick-up for example), or a page with the selected images may be printedwhich the user can insert into an actual frame that matches the digitalframe depicted on the display.

At step 905, after the user has identified favorite or primary images onthe computer system, the algorithm determines if the number of primaryimages identified is greater then the number of openings in thetemplate. If so, then at step 1240 a subset, equal to the number ofopenings, of the favorite images whose image attributes satisfy requiredtemplate attributes and template opening attributes, if any, areautomatically selected and then in step 910, the template, together withthe subset favorite images disposed in the openings is composed, and isdisplayed to the user for user approval at step 915. If an option tobypass user approval 912 is active then the template and automaticallyselected subset of the favorite images is output at step 916. Otherwise,if the user approves the template and images, the user indicatesacceptance to the computer system of the template and images asdisplayed and the output step 916 commences. If the user approval stepresults in a non-acceptance by the user then the system proceeds back tostep 910 to repeat automatic image selection as described above, exceptthat the output of the automatic image selection step will be selectedto be different than any previous output. As usual, programmedpreference is given to higher scoring images that best match requiredtemplate opening attributes, but if the user does not accept these, thenlower scoring images will be used. If user control option 914 isactivated then the user has the option to specify favorite imageswithout reactivating the automatic selection step 1240. This can beperformed by presenting to the user the template with current selectedimages displayed therein, and an option to replace the current imageswith other user selected images from an image data base. The user'sfavorite image selections are then used in the output step 916. At thispoint it is an optional programming step to limit the user option toonly those images that are favorites, selected at step 830, or topresent the whole user's image data base for selection.

If the number of favorite images is determined not to be greater thanthe number or template openings at step 905, then at step 925 thecomputer system determines, according to the programmed algorithm, ifthe number of favorite images indicated by the user is equal to thenumber of template openings at step 925. If so, in step 1230, eachfavorite image is assigned the optimal window such that the overalltemplate score is maximized, then the images are placed in said windowopenings in step 940, and then output step at step 946. If an optionaluser control step 942 is active, the template and images disposedtherein are displayed to the user to enable the user to select differenttemplates or images, or to rearrange the images displayed. After theuser has completed his selections, the output step 946 is activated.

If, at step 925, the computer system determines that the number offavorite images is not equal to the number of template openings, thealgorithm selects additional images to compliment the favorite imagesusing step 1220 which is described above. At step 930 theseautomatically selected images will be digitally placed into thepreviously selected digital template. At step 935 the template andselected images will be displayed to the user on a screen or monitor andthe user will have the option to accept the image product at step 935.This embodiment of the present invention includes an option 934 which,when activated, allows the user to bypass review of the image productand go directly to output step 936. If the user accepts the displayedimage product, or if the review bypass occurs, the image enhancedproduct is output at step 936. If the user does not accept the outputproduct as displayed at step 935, then the system repeats an automaticimage selection using step 1220, but outputs a template plus images thatis different from any previous outputs. Alternatively, a user controloption 932 can be executed at this point wherein the user performs hisown selection step instead of the step 1220 performed automatically bythe computer, and the user's selection is then output at 936. The outputstep 936, as do each of the other output steps described above, operatesaccording to the type of image enhanceable product that the user hasselected. For example, and not by way of limitation, if a frame has beenselected, then the output can involve a frame with selected images beingdelivered to the user (home delivery or retail outlet pick-up forexample), or a page with the selected images formatted to match thetemplate openings may be printed which the user can insert into anactual frame that matches the digital frame depicted on the display. Asdescribed above, various image enhanceable products may be producedusing the inventive methods described herein and the provision of theseproducts can be completed at the output step in a variety of ways.

Referring to FIG. 7, if the template is to be selected entirely by thesystem we arrive at step 770, and we arrive at step 780 if some of thosetemplate attributes are manually selected by the user. Either way, wearrive at step 1010 of FIG. 10 and the template selection or templatecompletion algorithm begins. If the user has not yet provided images,either in a local computer accessible database, a network accessibledatabase or on a portable or other storage device, then at step 1020 thesystem will prompt the use to identify where images can be accessed. Theprompt may or may not occur if a data base has previously beenidentified to the system or if a user has coupled a portable or otherimage storage device to the system.

If manual mode is enabled, the user may decide, at step 1005, tomanually select one or more favorite images at step 1030 which are thenpassed to step 1040. If manual mode is disabled, the users images areautomatically selected 1040. The highest scoring template is chosen 1040by pairing the image set with the available templates in the system. Inthis instance, the score is not associated solely with the template,rather, it is a score generated by compatibility computation as betweenimage attributes and the required image attributes defined by each ofthe window recipes in the template recipe. If a user has selected someof the template options, these are not modified by the algorithm. If auser has picked favorite images, these images, or a subset of theseimages, can be required by programming to be included in the final userenhanceable product and the highest scoring template is selected basedon these favorite images being included in the template. Upon completionof template selection in step 1040, if automatic mode 1060 is enabled,the final template is selected, 1070. If automatic option is not turnedon, the user has the ability to override 1050 the template selectionprocess until he or she is satisfied. Once the user is satisfied, thefinal template is selected 1070 and the system now needs to fill thistemplate with images and migrates to decision 1125 in FIG. 11.

With regard to FIG. 11, the user has already been asked and optionallyselected their favorite images. The template was chosen by thealgorithm, or optionally overridden by manual controls by the user. Allthat is left to do is assemble the final product. If favorites wereselected, decision 1125 then uses strategies similar to step 930 in FIG.9, previously described. If no favorite images were selected we utilizeimage selection step 1210 and formatting in step 1130. If automatic mode1150 is enabled, the final user enhanceable product is assembled,delivered, printed and/or otherwise outputted 1160. If automatic optionis not turned on, the user has the ability to override 1140 the choicesmade by the image selection process until he or she is satisfied. Oncethe user is satisfied, the final user enhanceable product is assembled,delivered, printed and/or otherwise outputted 1160.

Regarding the automatic template selection process as shown in FIG. 7,further enhancements and modifications can be achieved with theseembodiments of the present invention as follows. For example, eachtemplate can be evaluated across multiple criteria such as size,orientation, number of window openings, theme, color, and texture. Uponcompletion of the automated template selection step 1070, the templatewith the highest score is the template that is chosen for the userenhanceable product. As described above, all template settings areinitially set to defaults at step 720. These default values can be setby the product manufacturer, individual retailer who has installed theimage enhanceable product tool described herein, or by individualproduct line requirements. For example, it is possible to have two sideby side kiosks in a store. One kiosk can be programmed with a “FamousPerson” Collection version of the software described herein, while theother is a generic unbranded version. If user interactive mode 705 isdisabled or if a user chooses at step 715 not to manually select atemplate, the default options 720 are passed at step 770 on to theautomatic template selection module starting with FIG. 10 at startselect images 1010.

If a user decides at step 715 to choose their own template, the user canspecify the exact template by scanning a template UPC barcodecorresponding to a user enhanceable product available at a retailer thathas installed a compatible kiosk, or itemized menu or the user cannarrow their choices using an interactive interface at step 740 to helpchoose all the template options. If the user does not specificallyprovide the exact template or does not provide all template options 725,there is still some ambiguity as to which template that will be used. Insuch a case, the default template options 720 are superseded by the userdefined options 760 and passed 780 on to the automatic templateselection module starting with FIG. 10 at start select template 1010.

If the user has provided all the template options at step 725 thisresults in a specific template to be used 750. We then enter the imageselection phase starting at 810 in FIG. 8. If no source of input imagescan be automatically determined, the user is prompted for such images820, for example, by displaying a request on a display screen, thoughother forms of prompting are possible, including an audible signal. Ifmanual mode is enabled, the user may decide to manually select one ormore favorite images at step 805. If manual mode is disabled, the user'sprimary images are automatically selected at step 1210. Before thedetails of automatic image selection are described, we will firstdescribe the process of the user selecting favorite images at step 830and the resulting program execution.

The selection of favorite images 830 involves the user previewing, forexample, thumbnail size images on the display screen and markingfavorite images. The user can select a number of images for n templateopenings in the previously selected template, where n is equal to thenumber of window openings in the product. For each image that is split,n is decremented by 1 for each window that the split image occupies. Tosimplify the description of the algorithm in this embodiment of theinvention, we will assume there is no image splitting. In addition toselecting their favorite images, the user is given the option ofselecting their overall favorite or primary image. If a primary image isselected, it generally will be displayed in a prominent fashion on theuser enhanceable product, and other template opening required imageattributes can be based upon the primary image attributes (color,sharpness, person ID, scene content, etc.).

At step 905, FIG. 9, the user has selected their n favorite images. Thefirst decision 905 is if the number of favorite images selected isgreater than the number of window openings in the user enhanceableproduct. If n<=m at step 905, the next decision 925 is to determine ifn, the selected images, is equal to m, the number of window openings. Ifso, the system then automatically determines which image goes in whichwindow opening 940.

The procedure performed at 940 is as follows. The algorithm analyzes theimages and determines what attributes they have either by accessing themetadata or semantic information extracted from the image or both. Boththe metadata and semantic information can be calculated and stored withthe image, or in a separate file. The fitness score of each image toeach window opening is calculated and then the program initializes thetemplate by assigning the highest scoring image to each window opening.If one image has the same score on two window openings, the priority isassigned to the lowest numbered window and the image is assigned to thatopening. This presumes that the windows in the template are numbered.For example, window 0 (“W0”) is stored, by default, as the highestpriority window, window 1 (“W1”) is next highest, and so forth. Windowsfor each template can be assigned numbers internally in a template fileor in a window file. If two different images score identically high inone window opening, the image can also be assigned based on metadatafields that record the number of times an image was accessed, randomly,or by another selection method. An overall sum total score for theentire template mapping is computed and is the baseline starting pointscore before the algorithm iterates through the list of templateopenings for scoring other image combinations in the windows todetermine if the baseline is exceeded. If another combination of imagesin the window openings results in an overall score higher than thebaseline score, then that combination becomes our new highest scoringcombination and we update our baseline score accordingly. We proceeduntil another combination again exceeds the current score or untiliterations are programmed to end.

The algorithm updates the overall sum total score after each iterationthrough the list of template openings and images and can run through allpossible combinations, until a set time period expires, or until someother programmed stop. The program can iterate through all images, or asubset, comparing a candidate image's attributes with some or all of thetemplate openings' required attributes and computing new candidate imagescores. When a candidate image's new fitness score is higher than acurrently assigned image's score the algorithm enters into a potentialswap condition.

Once the potential swap condition occurs, the image window's existingimage can be scored against the candidate image's currently assignedimage window and if both images score higher for both image windows thenthe swap can be selected. Swapping only happens if both images eachscore higher in the other opening and increase the template's overallscore.

The algorithm will, at some point, contain a list of image windows H anda list of images I. A mapping of images to image windows where I₁ mapsto H₁, I_(n) maps to H_(n) results in a fitness measurement score. Eachimage to image window mapping will be scored based on image attributesand required image window's attributes weighting factor. The algorithmcan be selected to run for a computed number of iterations (number ofimages×number of holes or a subset thereof), until a predeterminedfitness score is achieved, until the improvement from iteration toiteration is below a predetermined threshold, or until some otherprogrammed cut off.

For each image window opening, the following are programming options:

The algorithm can swap the existing image I_(i) with every image I_(r)in the list of images and perform a weighted attributes fitnessmeasurement score calculation by adding up the attributes weights thatthe image satisfies and determine if the I_(r) is a better fit withimage window H_(i).

If, for example, I_(r) to H_(i) is a better fit then the program cancompare both images' (I_(i) and I_(r)) scores with their new possibleswapped image windows (H_(i) and H_(r)) to determine if the averageweighted attributes fitness measurement score is overall better. If theaverage fitness measurement score is better the two images' windowopenings are swapped, if it not, they are not swapped.

After the processing has occurred for each image window opening in alist, a total score for all image to image window mappings will becomputed by summing the entire list of image scores and then comparingwith the previous total score. If the newer score is better than theprevious solution's score, the previous can be replaced with the newercandidate solution.

If we define imageList as the list of input images and imageWindowListas the list of window openings, the following algorithm can be used toiteratively find the optimal image to window opening configuration whenthe number of images is the same as the number of window openings:

PerformFitting( imageList, imageWindowList) OverallTemplatescore = 0;AssignImagesToImageWindows ( ); FOR each image Window in imageWindowListFOR each image in imageList Calculate newImageWindowScore IFnewImageWindowScore > imageWindowScore CheckForSwap(imageWindow,candidateImageWindow) ENDIF END FOR END FOR FOR each image Window inimageWindowList ADD the imageWindow.image.score to OverallTemplatescoreEND FOR RETURN OverallTemplatescore END PerformFittingCheckForSwap(imageWindow, candidateImageWindow) currentScore =Calculate2WindowScore(imageWindow, currentimage, candidateWindow,candidateimage) newScore = Calculate2WindowScore(imageWindow,candidateimage, candidateWindow, currentimage) IF newScore >currentScore THEN SwapImage(imageWindow, candidateImageWindow); ENDIFEND CheckForSwap AssignImagesToImageWindows( imageList, imageWindowList) N = NumberImages; FOR 1=1:N WindowScore[i] =ScoreImage(imageList[i],_(,) imageWindow[i]) END FOR EndAssigImagesToImageWindows ScoreImage( image,_(,) imageWindow) imageScore= 0 FOREACH entry in recipe imageScore += PerformMeasurement(image,recipeEntry); END FOR RETURN imageScore END ScoreImage

Upon completion of step 940, if automatic mode 944 is enabled, the finaluser enhanceable product can be assembled, delivered, printed, etc. 946.If automatic option is not turned on, the user has the ability tooverride 942 the choices made by the image selection process until he orshe is satisfied. For example, the user may not like the automaticrearrangement of images done by the automatic algorithm, or it ispossible the user may want to select a new image and insert it into theproduct Once the user is satisfied, the final user enhanceable productcan be assembled, delivered, and/or printed 946 as explained above.Referring back to step 905 of FIG. 9, at this point the user hasselected their n favorite images. If the number of favorite images n isgreater than the number of window openings m in the user enhanceableproduct the software will select only the top m images. To select thetop m images, the metadata and semantic information necessary to fulfilleach template opening recipe is automatically extracted from the nimages. The individual opening recipes are evaluated for each image ateach window opening. As when n=m, such calculations involve a fitnessscore of each favorite image paired with each window opening, and thenthe summation of all window opening fitness scores comprises the overalltemplate score.

There are multiple ways to select the optimal image to window openingconfiguration. The simplest approach is to evaluate the score of eachimage to each window opening and then assign the highest scoring imageto each window opening. If one image has the same score on two windowopenings, a priority is assigned to each window opening and the image isassigned to the highest priority window opening. If two different imagesscore identically high in one window opening, the image can be assignedbased on metadata fields that record the number of times an image wasaccessed, randomly, or by another programmable procedure.

More sophisticated approaches can be used to maximize the overalltemplate score as a whole. For small m and n, it is reasonable toevaluate all combinations and select the template along with m imagesarranged having the overall highest fitness score. For large m or n, alinear regression mapping m images to n window openings, where m>n, suchas linear least squares or more complex non-linear or iterativeapproaches can be used, as described below.

Once the top m images are selected, and if automatic mode 912 isenabled, the final user enhanceable product is output 916. If automaticoption is not turned on, the user has the ability to override 914 thechoices made by the image selection process until he or she issatisfied, at which point the final user enhanceable product 916 isoutput or assembled, as the case requires.

The third possibility in FIG. 9 is when the number of favorite images nis smaller than the number of window openings m. In this case, theprogram must search through the user's images and select the top scoring(m−n) images to go along with the n favorites to fill the m windowopenings. If n and m are small, all possibilities can be tried, and thehighest scoring template with the corresponding m favorites and n−mautomatically selected images will be used in step 930. Often however,the user image selection can be large, often requiring the analysis ofhundreds, and sometimes thousands of images to select the m−n images. Inthis case, the decision of choosing the highest scoring images toachieve the highest scoring template can be iterative. To speedexecution, this process is systematic. If a favorite image is selectedit is first evaluated and inserted into the highest scoring windowopening. Then other user selected favorite images are inserted into theother window openings such that the highest partial template score isachieved. This leaves (m−n) window openings to fulfill with the user'simage collection.

The simplest approach is to evaluate each candidate image to each of theremaining m−n window openings and pick the highest scoring image foreach window opening, resolving ties by utilizing a priority with eachwindow opening. When the image selection to pick from is obtrusivelylarge, collecting semantic information for each image can be quite timeconsuming. A preferred embodiment is to only use recorded metadata toevaluate the fitness of each image as this is fast and can be done onany number of images in a quick fashion. The top x %, or the top p,where x or p is a user defined parameter, images are then selected forfurther interrogation, by evaluating the semantic information along withthe metadata when evaluating the fitness score of each image. Anotherembodiment of the present invention is to analyze images in ahierarchical fashion. In the first layer, all images are evaluated bymetadata alone. The top x % or p images are passed to the second round.In the second round, simpler, or fast semantic algorithms, such as coloror histogram semantics are evaluated in the second round, where the topy % or q images are passed to the third round. In the third round,slightly more complex semantic algorithms such as face detection areevaluated and the top z % or r images are passed to the fourth round. Inthe fourth, or final round, the most complex semantic algorithms such asfacial recognition, custom object detection are performed. This can beextended to any number of rounds. As we go from the first to last round,the number of candidate images shrinks, but the computational complexityof each analysis increases.

Additional programming features can be optionally implemented asfollows:

In step 770, all template selection items were completed by default. Instep 780, some of those default selection items were overridden by theconsumer. For example, for step 780, all template options may be thesame as in step 770 except that the user may override the number ofwindow openings by reducing the total to two. Thus, in FIG. 10, at step1010, the program begins to determine which template will be selected,given the selection of user imagery. For example, if all user images areidentified as fall scenes, then a fall oriented template would score thehighest. If no source of input images can be automatically determined bythe computer system, the user is prompted for such images 1020. Ifmanual mode is enabled, the user may decide to manually select one ormore favorite images 1005. If manual mode is disabled, the users imagesare automatically selected 1040. The selection of favorite images 1030can involve the user previewing thumbnail size images on a displayscreen and marking favorite images. The user can select as many imagesas desired, however, selection can be limited by program to a number z,where z is equal to the number of window openings in the product withthe most window openings, in this example z=2. Therefore, creating andstoring a number of templates having the same theme but each with adifferent number of windows can better accommodate user preferences.Once again, we will assume there is no image splitting. In addition toselecting favorite images, the user is given the option of selectingtheir overall favorite, primary, or “hero” image.

The highest scoring template is chosen 1040 by pairing the image setwith the available templates in the system. If a user has pickedfavorite images, these images, or a subset of these images can berequired by programming to be included in the final user enhanceableproduct. If a “hero” image is marked, the template selected must have a“hero” window opening that is highly compatible with the “hero” imageselected by the user. When scoring each template, each template isinitially set to zero. It is possible for a manufacturer or retailer, tobias the creation of some templates over others by purposelyinitializing such templates at a higher score. For example, templatesthat generate more royalties, or that are faster to process, or thatyield higher quality products may be preset at higher levels toencourage their usage over less desirable templates. Similarly,templates that are not desired may be given negative scores. If a singletemplate is always to be used, it is given a score of positive 1000. Ofcourse, if two or more templates are prescored at 1000, the one with thehighest score after image evaluation is chosen. In the case of a tie,all templates can be assigned a numeric priority, and the template withthe highest priority is chosen.

The decision of choosing the highest scoring template can be iterative.Each template in the system is evaluated one at a time across allimages. To speed execution, this process is hierarchical. If a heroimage is selected it is first evaluated across all templates. The top x%, where x is an adjustable/selectable parameter, templates areevaluated further, and the bottom (100−x) % templates are discarded.After the hero selection, the favorite images are scored in eachtemplate in each window opening. The highest scoring configuration foreach template is stored. If n>m (if user favorite image is greater thanwindow openings), then the selection is complete. If not, the programpicks the top y %, where y is an adjustable parameter, and templates areevaluated further, and the bottom (100−y) % templates are discarded.Once again, for the remaining window openings in each remainingtemplate, each image is evaluated and the top scoring templates arerecorded. This hierarchical approach greatly speeds up programexecution, especially when there are many templates, and many images toevaluate.

After the top scoring template is chosen, in automatic mode 1060, thistemplate is selected by the algorithm 1070 and it along with its chosenimages are passed to step 1125 in FIG. 11. If automatic mode isdisabled, the user is prompted for acceptance 1015 of the template. Ifthe user rejects this template, the rest of the templates are offered tothe user in order of decreasing score. Eventually, the user will selectone of the templates 1070 and it passes to step 1125 in FIG. 11.

Referring now to FIG. 13, a sample list of recorded metadata tagsobtained from image acquisition and utilization systems includingdigital stand-alone cameras, cell phone cameras, personal computers,digital picture frames, camera docking systems, imaging appliances,networked displays, and printers. Recorded metadata is synonymous withinput metadata and includes information recorded by an imaging deviceautomatically and from user interactions with the device. Standard formsof recorded metadata include; time/date stamps, location informationprovided by global positioning systems (GPS), nearest cell tower, orcell tower triangulation, camera settings, image and audio histograms,file format information, and any automatic images corrections such astone scale adjustments and red eye removal. In addition to thisautomatic device centric information recording, user interactions canalso be recorded as metadata and include; “Share”, “Favorite”, or“No-Erase” designation, “Digital print order format (DPOF), userselected “Wallpaper Designation” or “Picture Messaging” for cell phonecameras, user selected “Picture Messaging” recipients via cell phonenumber or e-mail address, and user selected capture modes such as“Sports”, “Macro/Close-up”, “Fireworks”, and “Portrait”. Imageutilizations devices such as personal computers running Kodak EasyShare™ software or other image management systems and stand alone orconnected image printers also provide sources of recorded metadata. Thistype of information includes print history indicating how many times animage has been printed, storage history indicating when and where animage has been stored or backed-up, and editing history indicating thetypes and amounts of digital manipulations that have occurred.

FIG. 14 contains a sample list of extracted metadata tags obtained fromanalysis of image content and existing recorded metadata tags. Extractedmetadata tags can be created by image acquisition and utilizationsystems including standalone digital cameras, cell phone cameras,personal computers, digital picture frames, camera docking systems,imaging appliances, networked displays, and printers. Extracted metadatatags can be created automatically when certain predetermined criteriaare met or from direct user interactions. An example of the interactionbetween extracted metadata and recorded metadata is using a cameragenerated image capture time/date stamp in conjunction with a user'sdigital calendar. Both systems can be co-located on the same device aswith a cell phone camera or can be dispersed between imaging devicessuch as a camera and personal computer camera docking system. A digitalcalendar can include significant dates of general interest such as:Cinco de Mayo, Independence Day, Halloween, Christmas, and the like andsignificant dates of personal interest such as; “Mom & Dad'sAnniversary”, “Aunt Betty's Birthday”, and “Tommy's Little LeagueBanquet”. Camera generated time/date stamps can be used as queries tocheck against the digital calendar to determine if any images werecaptured on a date of general or personal interest. If matches are madethe metadata can be updated to include this new extracted information.Further context setting can be established by including other extractedand recorded metadata such as location information and locationrecognition. If, for example, after several weeks of inactivity a seriesof images and videos are recorded on September 5^(th) at a location thatwas recognized as “Mom & Dad's House”. In addition the user's digitalcalendar indicated that September 5^(th) is “Mom & Dad's Anniversary”and several of the images include a picture of a cake with text thatreads, “Happy Anniversary Mom & Dad”. Now the combined extracted andderived metadata can automatically provide a very accurate context forthe event, “Mom & Dad's Anniversary”. With this context established onlyrelevant theme choices would be made available to the user significantlyreducing the workload required to find an appropriate theme. Alsolabeling, captioning, tagging, or blogging, can be assisted or automatedsince the event type and principle participants are now known to thesystem.

The content of image, video, and audio assets can be analyzed usingface, object, speech, and text identification and algorithms. The numberof faces and relative positions in a scene or sequence of scenes canreveal important details to provide a context for the images. Forexample a large number of faces aligned in rows and columns indicates aformal posed context applicable to family reunions, team sports,graduations, and the like. Additional information such as team uniformswith identified logos and text would indicate a “sporting event”,matching caps and gowns would indicate a “graduation”, and assortedclothing may indicate a “family reunion”, and a white gown, matchingcolored gowns, and men in formal attire would indicate a “WeddingParty”. These indications combined with additional extracted and derivedmetadata provide an accurate context that enables the system to selectappropriate images, provide relevant themes for the selected images, andto provide relevant additional images to the original image collection.

FIGS. 15A and 15B, illustrate, respectively, a top view and a sectionelevation view of another, non-limiting example of an image enhanceableproduct 1500. In this embodiment, image enhanceable product 1500comprises a conventional matte and frame combination 1518 having anexternal structural frame 1524, with a light transmissive area 1504 andan internal area 1505 that is adapted to hold a printed image. In theembodiment of FIGS. 15A and 15B, the framing matte includes a pluralityof windows 1530, 1532 and 1534 that allow light to pass through soimages disposed in region 1505 that are aligned with windows 1530, 1532and 1534 can be seen outside of the framing matte while other portion ofregion 1505 are blocked from view. In FIG. 15A printed image sheet isdepicted at 1510 to illustrate the placement location of the printedsheet. In the embodiment that is illustrated, it is assumed that windows1530, 1532 and 1534 are essentially transparent. However, it will beappreciated that in various embodiments, windows 1530, 1532, and 1534can comprise transparent or semi-transparent materials that allow lightto pass therethrough in a modified form. For example, windows 1530,1532, and 1534 can filter, soften, or even selectively block portions oflight passing therethrough as may be desired. In certain embodiments,liquid crystal display or other active or semi-active light blockingmaterials can be used. Further, in certain embodiments, filtering can beperformed for artistic or aesthetic purposes, while in the same or otherembodiments, filtering can be protective such as where the filteringblocks forms of light that can damage images disposed therein.

In the embodiment of FIGS. 15A and 15B, internal area 1505 is also sizedand shaped to hold an optional backing support 1528, which can have, forexample, mounting structures (not shown) such as hook mountings and thelike defined therein to hold it in place in structure 1524. The printedimage template can be place on inside surface of backing 1528 and thenbacking 1528 is secured in framing matte 1500. In other embodiments,internal area 1505 can optionally be sized to hold a protection layersuch as a glass or other transparent or semitransparent sheet (notshown) of conventional design to protect, hold, and/or secure imagestherein.

PARTS LIST  22 housing  24 system  26 system  28 system  29 printer  30printer  32 I/O  34 processor  35 I/O  38 sensor  39 memory  40 storage 42 storage  44 storage  46 comm  48 memory  50 interface  52 memory  54system  56 I/O  58 I/O  58a I/O  58b I/O  58c I/O  66 I/O  68 I/O  68aI/O  68b I/O  70 system  72 user  201 sensor  300 image enhanceable item 302 image receiving area  306 window area  308 window area  310 windowarea  312 inter-window area  320 decorative background image  330decorative background image  610 image  620 image  630 image  640 image 615 label  625 label  635 label  645 label  640 image  660 image  670image  705 step  710 step  715 step  720 step  725 step  730 step  740step  750 step  760 step  770 step  780 step  805 step  810 step  815step  820 step  830 step  840 step  850 step  860 step  870 step  905step  910 step  912 step  914 step  915 step  916 step  925 step  930step  932 step  934 step  935 step  936 step  940 step  942 step  944step  945 step  946 step 1005 step 1010 step 1015 step 1020 step 1030step 1040 step 1050 step 1060 step 1070 step 1105 step 1125 step 1130step 1140 step 1150 step 1160 step 1210 step 1220 step 1230 step 1240step 1500 image enhanceable item 1510 image receiving area 1518inter-window area 1530 window area 1532 window area 1534 window area1504 light transmissive area 1505 internal area 1524 structural frame1528 backing area 1530 window 1532 window 1534 window

The invention claimed is:
 1. A computer system comprising: a circuit forcoupling the computer system to a digital image database, each of thedigital images including image attributes associated therewith; aworking memory coupled to the circuit for storing images obtained fromthe database and for storing a data base of digital templates, each ofthe templates including a plurality of windows for displaying one ormore of the digital images, wherein each of the templates includes afirst window, corresponding to a primary window for displaying a primarydigital image having first required image attributes and weightingfactors associated with each image attribute, at least one secondwindow, corresponding to a secondary window for displaying a secondarydigital image having second required image attributes and weightingfactors associated with each image attribute, and an initial bias score;a processor for automatically extracting image attributes from theworking memory that are associated with a selected one of the imagesstored therein, for evaluating the extracted image attributes, forweighting the evaluated image attributes using the correspondingweighting factor, for computing a fitness score for each template basedat least upon the initial bias score and the weighted image attributescorresponding to displaying the primary image in the first window forselecting templates where the first required image attributes aresatisfied by the weighted image attributes for further evaluation, forupdating the fitness score for each selected template based upon theweighted image attributes corresponding to displaying the secondaryimages in the remaining of the plurality of windows by evaluatingcombinations of placing a subset of the secondary images in theremaining of the plurality of windows so that the second required imageattributes are satisfied by the weighted image attributes and selectingthe combination that results in the highest fitness score for eachtemplate, and for selecting the templates with the highest fitnessscores; and a display screen to present at least one of the highestscoring selected templates to a user.
 2. The computer system accordingto claim 1 wherein the digital image database is contained in a portablememory device.
 3. The computer system according to claim 1 wherein thedigital image database is contained in a network accessible storagedevice.
 4. The computer system according to claim 1 wherein the displayscreen includes a plurality of windows, and wherein the selectedtemplate and the at least one other digital image are displayed inseparate windows.
 5. The computer system according to claim 1, whereinthe selected template and the at least one other digital image aredisplayed as a list of template and image file names.
 6. The computersystem according to claim 1 further comprising a printer coupled to thecomputer system for printing the selected template and the at least oneother digital image.
 7. The computer system according to claim 6 whereinthe printed selected template and the at least one other digital imagecomprises one of the following image display products: an album page; alabel; a sticker.
 8. The computer system according to claim 1 furthercomprising user controls for permitting a user to modify or replace theat least one other digital image.
 9. The computer system according toclaim 1 further comprising user controls for permitting a user torelocate at least one of said windows within the digital template. 10.The computer system according to claim 1 further comprising usercontrols for storing as a computer accessible file the selected templateand the at least one other digital image.
 11. The computer systemaccording to claim 10 further comprising a circuit for transmitting thecomputer accessible file over a communication network.
 12. The computersystem according to claim 1 further comprising means for calculating acompatibility score for images to determine which image best satisfiesimage attribute requirements for a particular window.